-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63109.79 Mb (on task=120), Smallest =   62839.11 Mb (on task=0), Average =   63039.72 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1692.66 Mb (on task=0), Smallest =    1422.27 Mb (on task=120), Average =    1492.35 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.10 Mb
AllocMem:	 Largest =    1692.66 Mb (on task=0), Smallest =    1422.27 Mb (on task=120), Average =    1492.35 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_028_z000p036/snap_028_z000p036.0.hdf5' on task=0 (contains 132044365 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  132014885  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      7726  (tot=     0000134208) masstab=0
Type 3 (bulge):    21754  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.967907

Setting next time for line of sight file to Time_next= 0.970624

Setting next time for snipshot file to Time_next= 0.967365

Setting next time for fof output to Time_next= 0.967365
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=9.7805 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=102337
After=103097
NTopleaves= 90210  NTopnodes=103097 (space for 465370)
gravity work-load balance=1.00032   memory-balance=1.00032   SPH work-load balance=0
iter=0 exchange of 2111413074 particles (ret=0)
domain decomposition done. (took 38.8006 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5835 MByte in top-level domain structure

Allocated 7296.2 MByte for BH-tree, and 0.344124 Mbyte for top-leaves.  (presently allocted 25777.3 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.2 Mbyte (on task=74), Smallest = 26119.7 Mbyte, Average = 26120.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  74 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  74 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  74 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  74 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  74 2972 0          TopNodes      5.1127   18480.7720  domain_allocate()/domain.c/570
  74 2973 0   DomainNodeIndex      0.3441   18481.1162  force_treeallocate()/forcetree.c/4800
  74 2974 0        Nodes_base   3737.4527   22218.5689  force_treeallocate()/forcetree.c/4803
  74 2975 0     Extnodes_base   3114.5439   25333.1128  force_treeallocate()/forcetree.c/4811
  74 2976 0          Nextnode    222.2989   25555.4116  force_treeallocate()/forcetree.c/4820
  74 2977 0            Father    221.9056   25777.3172  force_treeallocate()/forcetree.c/4827
  74 2978 0           Ngblist     63.4222   25840.7394  density()/density.c/690
  74 2979 0              Left    126.8443   25967.5837  density()/density.c/692
  74 2980 0             Right    126.8443   26094.4281  density()/density.c/693
  74 2981 0    DataIndexTable      7.0175   26101.4456  density()/density.c/738
  74 2982 0      DataNodeList     18.7134   26120.1591  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.000.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.049.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.3 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=93697
After=103921
NTopleaves= 90931  NTopnodes=103921 (space for 465370)
gravity work-load balance=1.00061   memory-balance=1.00061   SPH work-load balance=0
iter=0 exchange of 2110693532 particles (ret=0)
domain decomposition done. (took 31.1062 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5458 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.8 MB)
Tree construction.

Start linking particles (presently allocated=26538.1 MB)
links on local processor done (took 185.768 sec).
Marked=0245373498 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26862.1 MB) 
have done 0003045081 cross links (processed 0245373498, took 32.6682 sec)
have done 0001343284 cross links (processed 0206850460, took 34.6585 sec)
have done 0000980441 cross links (processed 0182025888, took 30.1858 sec)
have done 0000673086 cross links (processed 0133370893, took 26.4316 sec)
have done 0000434469 cross links (processed 0074251142, took 22.257 sec)
have done 0000262769 cross links (processed 0037821708, took 16.611 sec)
have done 0000158651 cross links (processed 0028504984, took 11.7032 sec)
have done 0000097962 cross links (processed 0018885621, took 17.3066 sec)
have done 0000062045 cross links (processed 0006416174, took 4.29029 sec)
have done 0000036360 cross links (processed 0003941982, took 2.1088 sec)
have done 0000020665 cross links (processed 0001944789, took 1.88345 sec)
have done 0000014230 cross links (processed 0002357905, took 4.66762 sec)
have done 0000007989 cross links (processed 0001660767, took 4.21382 sec)
have done 0000003672 cross links (processed 0000313465, took 1.50911 sec)
have done 0000001310 cross links (processed 0000019765, took 0.179111 sec)
have done 0000000494 cross links (processed 0000006997, took 0.176079 sec)
have done 0000000193 cross links (processed 0000002611, took 0.174823 sec)
have done 0000000085 cross links (processed 0000000964, took 0.199936 sec)
have done 0000000021 cross links (processed 0000000349, took 0.180684 sec)
have done 0000000006 cross links (processed 0000000081, took 0.174163 sec)
have done 0000000000 cross links (processed 0000000021, took 0.177363 sec)
Local groups found.

group finding took = 398.05 sec
Start finding nearest dm-particle (presently allocated=26538.1 MB)
still finding nearest... (presently allocated=26805.5 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.438868 sec
compiling local group data and catalogue took = 10.1965 sec

Total number of groups with at least 32 particles: 1257012
Largest group has 154502984 particles.
Total number of particles in groups: 1324428597

group properties are now allocated.. (presently allocated=19505.9 MB)
computation of group properties took = 0.706544 sec
start global sorting of group catalogues number 28
LargestGroup = 120441.101250; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.222383 sec
Group catalogues globally sorted. took = 8.68357 sec
starting saving of group catalogue number 28
data//groups_028_z000p036
Group catalogues number 28 saved. took = 15.4022 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 11.668 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 3.32528 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 12.4004 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  35 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  35 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  35 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  35 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  35 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  35 2972 0          TopNodes      5.1536   18480.8129  domain_allocate()/domain.c/570
  35 2973 0         FOF_PList    380.5976   18861.4105  fof_fof()/fof.c/249
  35 2974 0         FOF_GList    634.3294   19495.7400  fof_fof()/fof.c/337
  35 2975 0             Group     10.1902   19505.9302  fof_fof()/fof.c/380
  35 2976 0   DomainNodeIndex      0.3469   19506.2771  force_treeallocate()/forcetree.c/4800
  35 2977 0        Nodes_base   3737.5281   23243.8052  force_treeallocate()/forcetree.c/4803
  35 2978 0     Extnodes_base   3114.6068   26358.4120  force_treeallocate()/forcetree.c/4811
  35 2979 0          Nextnode    222.3020   26580.7140  force_treeallocate()/forcetree.c/4820
  35 2980 0            Father    221.9056   26802.6196  force_treeallocate()/forcetree.c/4827
  35 2981 0           Ngblist     63.4329   26866.0526  subfind_density()/subfind_density.c/78
  35 2982 0         Dist2list    126.8659   26992.9184  subfind_density()/subfind_density.c/79
  35 2983 0    DataIndexTable     21.8182   27014.7366  subfind_density()/subfind_density.c/87
  35 2984 0      DataNodeList     58.1818   27072.9184  subfind_density()/subfind_density.c/89
  35 2985 0              Left    126.8659   27199.7843  subfind_density()/subfind_density.c/91
  35 2986 0             Right    126.8659   27326.6502  subfind_density()/subfind_density.c/92
  35 2987 0              Todo     15.8582   27342.5084  subfind_density()/subfind_density.c/93
  35 2988 0             DM_Vx    126.8659   27469.3743  subfind_density()/subfind_density.c/95
  35 2989 0             DM_Vy    126.8659   27596.2402  subfind_density()/subfind_density.c/96
  35 2990 0             DM_Vz    126.8659   27723.1061  subfind_density()/subfind_density.c/97
  35 2991 0    	  DensDataGet    163.0060   27886.1120  subfind_density()/subfind_density.c/161
----------------------------------------------------------------------------------------

Task=35: Not enough memory in mymalloc_fullinfo() to allocate 116.364 MB for variable '	  DensDataIn' at subfind_density()/subfind_density.c/line 163 (FreeBytes=113.888 MB).
task 35: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 35
-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63110.24 Mb (on task=120), Smallest =   62841.80 Mb (on task=0), Average =   63040.12 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1689.96 Mb (on task=0), Smallest =    1421.82 Mb (on task=120), Average =    1491.96 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.11 Mb
AllocMem:	 Largest =    1689.96 Mb (on task=0), Smallest =    1421.82 Mb (on task=120), Average =    1491.96 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_027_z000p073/snap_027_z000p073.0.hdf5' on task=0 (contains 136190228 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  135343139  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      6784  (tot=     0000134208) masstab=0
Type 3 (bulge):   840305  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.9443

Setting next time for line of sight file to Time_next= 0.93275

Setting next time for snipshot file to Time_next= 0.933597

Setting next time for fof output to Time_next= 0.933597
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=9.03937 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=102577
After=103281
NTopleaves= 90371  NTopnodes=103281 (space for 465370)
gravity work-load balance=1.0004   memory-balance=1.0004   SPH work-load balance=0
iter=0 exchange of 2111062005 particles (ret=0)
domain decomposition done. (took 29.6375 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5751 MByte in top-level domain structure

Allocated 7296.23 MByte for BH-tree, and 0.344738 Mbyte for top-leaves.  (presently allocted 25777.4 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.2 Mbyte (on task=80), Smallest = 26119.7 Mbyte, Average = 26120.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  80 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  80 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  80 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  80 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  80 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  80 2972 0          TopNodes      5.1218   18480.7812  domain_allocate()/domain.c/570
  80 2973 0   DomainNodeIndex      0.3447   18481.1259  force_treeallocate()/forcetree.c/4800
  80 2974 0        Nodes_base   3737.4695   22218.5954  force_treeallocate()/forcetree.c/4803
  80 2975 0     Extnodes_base   3114.5580   25333.1534  force_treeallocate()/forcetree.c/4811
  80 2976 0          Nextnode    222.2996   25555.4530  force_treeallocate()/forcetree.c/4820
  80 2977 0            Father    221.9056   25777.3586  force_treeallocate()/forcetree.c/4827
  80 2978 0           Ngblist     63.4268   25840.7854  density()/density.c/690
  80 2979 0              Left    126.8537   25967.6391  density()/density.c/692
  80 2980 0             Right    126.8537   26094.4928  density()/density.c/693
  80 2981 0    DataIndexTable      7.0175   26101.5103  density()/density.c/738
  80 2982 0      DataNodeList     18.7134   26120.2237  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.049.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.101.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.4 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=93801
After=104049
NTopleaves= 91043  NTopnodes=104049 (space for 465370)
gravity work-load balance=1.00052   memory-balance=1.00053   SPH work-load balance=0
iter=0 exchange of 2112889467 particles (ret=0)
domain decomposition done. (took 36.1682 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.54 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.8 MB)
Tree construction.

Start linking particles (presently allocated=26538.7 MB)
links on local processor done (took 162.577 sec).
Marked=0239216319 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26862.8 MB) 
have done 0003010124 cross links (processed 0239216319, took 34.1968 sec)
have done 0001342180 cross links (processed 0201105963, took 40.1894 sec)
have done 0000978317 cross links (processed 0175136247, took 36.0128 sec)
have done 0000668231 cross links (processed 0115621325, took 21.2164 sec)
have done 0000439927 cross links (processed 0075088811, took 19.2415 sec)
have done 0000279013 cross links (processed 0044439929, took 18.9017 sec)
have done 0000173267 cross links (processed 0024609254, took 10.639 sec)
have done 0000116425 cross links (processed 0017196631, took 9.69427 sec)
have done 0000084117 cross links (processed 0009771423, took 7.95905 sec)
have done 0000055154 cross links (processed 0006799377, took 4.61258 sec)
have done 0000034358 cross links (processed 0003912472, took 4.34757 sec)
have done 0000027288 cross links (processed 0004197228, took 4.19947 sec)
have done 0000023932 cross links (processed 0004374818, took 4.95058 sec)
have done 0000019200 cross links (processed 0003086566, took 4.54171 sec)
have done 0000013716 cross links (processed 0002346653, took 4.80148 sec)
have done 0000007297 cross links (processed 0001071055, took 4.26042 sec)
have done 0000002797 cross links (processed 0000055980, took 0.212943 sec)
have done 0000000886 cross links (processed 0000015546, took 0.202904 sec)
have done 0000000254 cross links (processed 0000003484, took 0.180183 sec)
have done 0000000078 cross links (processed 0000000988, took 0.17388 sec)
have done 0000000027 cross links (processed 0000000297, took 0.181544 sec)
have done 0000000005 cross links (processed 0000000098, took 0.363942 sec)
have done 0000000000 cross links (processed 0000000015, took 0.363131 sec)
Local groups found.

group finding took = 394.609 sec
Start finding nearest dm-particle (presently allocated=26538.7 MB)
still finding nearest... (presently allocated=26806.2 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.811654 sec
compiling local group data and catalogue took = 10.287 sec

Total number of groups with at least 32 particles: 1265418
Largest group has 152759837 particles.
Total number of particles in groups: 1314945874

group properties are now allocated.. (presently allocated=19507.3 MB)
computation of group properties took = 0.979257 sec
start global sorting of group catalogues number 27
LargestGroup = 119082.250185; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.224344 sec
Group catalogues globally sorted. took = 8.99917 sec
starting saving of group catalogue number 27
data//groups_027_z000p073
Group catalogues number 27 saved. took = 15.5917 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 7.89835 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 2.0587 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 7.16662 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  30 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  30 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  30 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  30 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  30 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  30 2973 0         FOF_PList    380.1121   18860.9314  fof_fof()/fof.c/249
  30 2974 0         FOF_GList    633.5202   19494.4516  fof_fof()/fof.c/337
  30 2975 0             Group     11.2019   19505.6535  fof_fof()/fof.c/380
  30 2976 0   DomainNodeIndex      0.3473   19506.0008  force_treeallocate()/forcetree.c/4800
  30 2977 0        Nodes_base   3737.5399   23243.5407  force_treeallocate()/forcetree.c/4803
  97 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  97 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  97 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  97 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  97 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  97 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  97 2973 0         FOF_PList    379.7353   18860.5545  fof_fof()/fof.c/249
  97 2974 0         FOF_GList    632.8922   19493.4467  fof_fof()/fof.c/337
  97 2975 0             Group     11.9614   19505.4081  fof_fof()/fof.c/380
  97 2976 0   DomainNodeIndex      0.3473   19505.7554  force_treeallocate()/forcetree.c/4800
  97 2977 0        Nodes_base   3737.5399   23243.2953  force_treeallocate()/forcetree.c/4803
  97 2978 0     Extnodes_base   3114.6165   26357.9118  force_treeallocate()/forcetree.c/4811
  97 2979 0          Nextnode    222.3025   26580.2143  force_treeallocate()/forcetree.c/4820
  97 2980 0            Father    221.9056   26802.1199  force_treeallocate()/forcetree.c/4827
  97 2981 0           Ngblist     63.2892   26865.4091  subfind_density()/subfind_density.c/78
  30 2978 0     Extnodes_base   3114.6165   26358.1572  force_treeallocate()/forcetree.c/4811
  30 2979 0          Nextnode    222.3025   26580.4597  force_treeallocate()/forcetree.c/4820
  30 2980 0            Father    221.9056   26802.3653  force_treeallocate()/forcetree.c/4827
  97 2982 0         Dist2list    126.5784   26991.9876  subfind_density()/subfind_density.c/79
  97 2983 0    DataIndexTable     21.8182   27013.8057  subfind_density()/subfind_density.c/87
  97 2984 0      DataNodeList     58.1818   27071.9875  subfind_density()/subfind_density.c/89
  97 2985 0              Left    126.5784   27198.5660  subfind_density()/subfind_density.c/91
  97 2986 0             Right    126.5784   27325.1444  subfind_density()/subfind_density.c/92
  30 2981 0           Ngblist     63.3520   26865.7174  subfind_density()/subfind_density.c/78
  30 2982 0         Dist2list    126.7040   26992.4214  subfind_density()/subfind_density.c/79
  30 2983 0    DataIndexTable     21.8182   27014.2396  subfind_density()/subfind_density.c/87
  97 2987 0              Todo     15.8223   27340.9667  subfind_density()/subfind_density.c/93
  97 2988 0             DM_Vx    126.5784   27467.5451  subfind_density()/subfind_density.c/95
  97 2989 0             DM_Vy    126.5784   27594.1236  subfind_density()/subfind_density.c/96
  30 2984 0      DataNodeList     58.1818   27072.4214  subfind_density()/subfind_density.c/89
  30 2985 0              Left    126.7040   27199.1254  subfind_density()/subfind_density.c/91
  30 2986 0             Right    126.7040   27325.8295  subfind_density()/subfind_density.c/92
  30 2987 0              Todo     15.8380   27341.6675  subfind_density()/subfind_density.c/93
  97 2990 0             DM_Vz    126.5784   27720.7020  subfind_density()/subfind_density.c/97
  97 2991 0    	  DensDataGet    139.4124   27860.1144  subfind_density()/subfind_density.c/161
  30 2988 0             DM_Vx    126.7040   27468.3715  subfind_density()/subfind_density.c/95
  30 2989 0             DM_Vy    126.7040   27595.0756  subfind_density()/subfind_density.c/96
  30 2990 0             DM_Vz    126.7040   27721.7796  subfind_density()/subfind_density.c/97
  97 2992 0  	  DensDataResult    104.5593   27964.6737  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=97: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=35.3263 MB).
task 97: endrun called with an error level of 812


  30 2991 0    	  DensDataGet    116.3543   27838.1339  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 97
  30 2992 0  	  DensDataResult     87.2657   27925.3997  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=30: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.6003 MB).
task 30: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 30
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  24 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  24 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  24 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  24 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  24 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  24 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  24 2973 0         FOF_PList    380.5250   18861.3442  fof_fof()/fof.c/249
  24 2974 0         FOF_GList    634.2083   19495.5525  fof_fof()/fof.c/337
  24 2975 0             Group     12.3007   19507.8532  fof_fof()/fof.c/380
  24 2976 0   DomainNodeIndex      0.3473   19508.2005  force_treeallocate()/forcetree.c/4800
  24 2977 0        Nodes_base   3737.5399   23245.7403  force_treeallocate()/forcetree.c/4803
 103 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 103 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 103 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 103 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 103 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 103 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 103 2973 0         FOF_PList    380.1741   18860.9933  fof_fof()/fof.c/249
 103 2974 0         FOF_GList    633.6235   19494.6168  fof_fof()/fof.c/337
 103 2975 0             Group     11.0795   19505.6963  fof_fof()/fof.c/380
 103 2976 0   DomainNodeIndex      0.3473   19506.0436  force_treeallocate()/forcetree.c/4800
 103 2977 0        Nodes_base   3737.5399   23243.5834  force_treeallocate()/forcetree.c/4803
 103 2978 0     Extnodes_base   3114.6165   26358.2000  force_treeallocate()/forcetree.c/4811
 103 2979 0          Nextnode    222.3025   26580.5025  force_treeallocate()/forcetree.c/4820
 103 2980 0            Father    221.9056   26802.4081  force_treeallocate()/forcetree.c/4827
 103 2981 0           Ngblist     63.3624   26865.7704  subfind_density()/subfind_density.c/78
 103 2982 0         Dist2list    126.7247   26992.4951  subfind_density()/subfind_density.c/79
 103 2983 0    DataIndexTable     21.8182   27014.3133  subfind_density()/subfind_density.c/87
 103 2984 0      DataNodeList     58.1818   27072.4951  subfind_density()/subfind_density.c/89
  24 2978 0     Extnodes_base   3114.6165   26360.3569  force_treeallocate()/forcetree.c/4811
 103 2985 0              Left    126.7247   27199.2198  subfind_density()/subfind_density.c/91
 103 2986 0             Right    126.7247   27325.9445  subfind_density()/subfind_density.c/92
 103 2987 0              Todo     15.8406   27341.7850  subfind_density()/subfind_density.c/93
 103 2988 0             DM_Vx    126.7247   27468.5097  subfind_density()/subfind_density.c/95
 103 2989 0             DM_Vy    126.7247   27595.2344  subfind_density()/subfind_density.c/96
 103 2990 0             DM_Vz    126.7247   27721.9591  subfind_density()/subfind_density.c/97
 103 2991 0    	  DensDataGet    116.5944   27838.5536  subfind_density()/subfind_density.c/161
 103 2992 0  	  DensDataResult     87.4458   27925.9994  subfind_density()/subfind_density.c/201
  24 2979 0          Nextnode    222.3025   26582.6594  force_treeallocate()/forcetree.c/4820
  24 2980 0            Father    221.9056   26804.5650  force_treeallocate()/forcetree.c/4827
  24 2981 0           Ngblist     63.4208   26867.9858  subfind_density()/subfind_density.c/78
----------------------------------------------------------------------------------------

Task=103: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.0006 MB).
task 103: endrun called with an error level of 812


  24 2982 0         Dist2list    126.8417   26994.8275  subfind_density()/subfind_density.c/79
  24 2983 0    DataIndexTable     21.8182   27016.6456  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 103
  24 2984 0      DataNodeList     58.1818   27074.8274  subfind_density()/subfind_density.c/89
  24 2985 0              Left    126.8417   27201.6691  subfind_density()/subfind_density.c/91
  24 2986 0             Right    126.8417   27328.5107  subfind_density()/subfind_density.c/92
  24 2987 0              Todo     15.8552   27344.3659  subfind_density()/subfind_density.c/93
  24 2988 0             DM_Vx    126.8417   27471.2076  subfind_density()/subfind_density.c/95
  24 2989 0             DM_Vy    126.8417   27598.0492  subfind_density()/subfind_density.c/96
  24 2990 0             DM_Vz    126.8417   27724.8909  subfind_density()/subfind_density.c/97
  24 2991 0    	  DensDataGet    138.9354   27863.8263  subfind_density()/subfind_density.c/161
  24 2992 0  	  DensDataResult    104.2016   27968.0279  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=24: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=31.9721 MB).
task 24: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 24
task 24: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 24:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 24
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
   1 2973 0         FOF_PList    380.0779   18860.8972  fof_fof()/fof.c/249
   1 2974 0         FOF_GList    633.4632   19494.3604  fof_fof()/fof.c/337
   1 2975 0             Group     12.4439   19506.8044  fof_fof()/fof.c/380
   1 2976 0   DomainNodeIndex      0.3473   19507.1517  force_treeallocate()/forcetree.c/4800
   1 2977 0        Nodes_base   3737.5399   23244.6915  force_treeallocate()/forcetree.c/4803
   1 2978 0     Extnodes_base   3114.6165   26359.3081  force_treeallocate()/forcetree.c/4811
   1 2979 0          Nextnode    222.3025   26581.6106  force_treeallocate()/forcetree.c/4820
   1 2980 0            Father    221.9056   26803.5162  force_treeallocate()/forcetree.c/4827
   1 2981 0           Ngblist     63.3463   26866.8625  subfind_density()/subfind_density.c/78
   1 2982 0         Dist2list    126.6926   26993.5551  subfind_density()/subfind_density.c/79
   1 2983 0    DataIndexTable     21.8182   27015.3733  subfind_density()/subfind_density.c/87
   1 2984 0      DataNodeList     58.1818   27073.5551  subfind_density()/subfind_density.c/89
   1 2985 0              Left    126.6926   27200.2477  subfind_density()/subfind_density.c/91
   1 2986 0             Right    126.6926   27326.9404  subfind_density()/subfind_density.c/92
 126 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 126 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 126 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 126 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 126 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 126 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 126 2973 0         FOF_PList    380.2586   18861.0778  fof_fof()/fof.c/249
 126 2974 0         FOF_GList    633.7643   19494.8421  fof_fof()/fof.c/337
 126 2975 0             Group     11.8940   19506.7361  fof_fof()/fof.c/380
 126 2976 0   DomainNodeIndex      0.3473   19507.0834  force_treeallocate()/forcetree.c/4800
 126 2977 0        Nodes_base   3737.5399   23244.6233  force_treeallocate()/forcetree.c/4803
 126 2978 0     Extnodes_base   3114.6165   26359.2398  force_treeallocate()/forcetree.c/4811
  96 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  96 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  96 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  96 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2987 0              Todo     15.8366   27342.7770  subfind_density()/subfind_density.c/93
   1 2988 0             DM_Vx    126.6926   27469.4696  subfind_density()/subfind_density.c/95
 126 2979 0          Nextnode    222.3025   26581.5423  force_treeallocate()/forcetree.c/4820
 126 2980 0            Father    221.9056   26803.4479  force_treeallocate()/forcetree.c/4827
  96 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  96 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  96 2973 0         FOF_PList    380.5842   18861.4035  fof_fof()/fof.c/249
   1 2989 0             DM_Vy    126.6926   27596.1623  subfind_density()/subfind_density.c/96
   1 2990 0             DM_Vz    126.6926   27722.8549  subfind_density()/subfind_density.c/97
 126 2981 0           Ngblist     63.3764   26866.8243  subfind_density()/subfind_density.c/78
 126 2982 0         Dist2list    126.7529   26993.5772  subfind_density()/subfind_density.c/79
 126 2983 0    DataIndexTable     21.8182   27015.3954  subfind_density()/subfind_density.c/87
  96 2974 0         FOF_GList    634.3070   19495.7105  fof_fof()/fof.c/337
  96 2975 0             Group     11.5620   19507.2724  fof_fof()/fof.c/380
  96 2976 0   DomainNodeIndex      0.3473   19507.6197  force_treeallocate()/forcetree.c/4800
   1 2991 0    	  DensDataGet    112.9717   27835.8266  subfind_density()/subfind_density.c/161
 126 2984 0      DataNodeList     58.1818   27073.5772  subfind_density()/subfind_density.c/89
 126 2985 0              Left    126.7529   27200.3300  subfind_density()/subfind_density.c/91
  96 2977 0        Nodes_base   3737.5399   23245.1596  force_treeallocate()/forcetree.c/4803
  96 2978 0     Extnodes_base   3114.6165   26359.7761  force_treeallocate()/forcetree.c/4811
   1 2992 0  	  DensDataResult     84.7288   27920.5554  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 126 2986 0             Right    126.7529   27327.0829  subfind_density()/subfind_density.c/92
 126 2987 0              Todo     15.8441   27342.9270  subfind_density()/subfind_density.c/93
 126 2988 0             DM_Vx    126.7529   27469.6799  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  96 2979 0          Nextnode    222.3025   26582.0786  force_treeallocate()/forcetree.c/4820
  96 2980 0            Father    221.9056   26803.9842  force_treeallocate()/forcetree.c/4827
  96 2981 0           Ngblist     63.4307   26867.4149  subfind_density()/subfind_density.c/78
  96 2982 0         Dist2list    126.8614   26994.2763  subfind_density()/subfind_density.c/79

Task=1: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.4446 MB).
task 1: endrun called with an error level of 812


  26 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  26 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  26 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  26 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  26 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  26 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  26 2973 0         FOF_PList    380.5284   18861.3476  fof_fof()/fof.c/249
  26 2974 0         FOF_GList    634.2139   19495.5616  fof_fof()/fof.c/337
  26 2975 0             Group     11.1303   19506.6919  fof_fof()/fof.c/380
  26 2976 0   DomainNodeIndex      0.3473   19507.0392  force_treeallocate()/forcetree.c/4800
  26 2977 0        Nodes_base   3737.5399   23244.5790  force_treeallocate()/forcetree.c/4803
  26 2978 0     Extnodes_base   3114.6165   26359.1956  force_treeallocate()/forcetree.c/4811
 126 2989 0             DM_Vy    126.7529   27596.4327  subfind_density()/subfind_density.c/96
 101 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 101 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 101 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 101 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 101 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 101 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 101 2973 0         FOF_PList    380.6063   18861.4255  fof_fof()/fof.c/249
 101 2974 0         FOF_GList    634.3438   19495.7693  fof_fof()/fof.c/337
 101 2975 0             Group     10.5015   19506.2708  fof_fof()/fof.c/380
 101 2976 0   DomainNodeIndex      0.3473   19506.6181  force_treeallocate()/forcetree.c/4800
 101 2977 0        Nodes_base   3737.5399   23244.1579  force_treeallocate()/forcetree.c/4803
 101 2978 0     Extnodes_base   3114.6165   26358.7745  force_treeallocate()/forcetree.c/4811
 101 2979 0          Nextnode    222.3025   26581.0770  force_treeallocate()/forcetree.c/4820
 101 2980 0            Father    221.9056   26802.9826  force_treeallocate()/forcetree.c/4827
 101 2981 0           Ngblist     63.4344   26866.4170  subfind_density()/subfind_density.c/78
  96 2983 0    DataIndexTable     21.8182   27016.0945  subfind_density()/subfind_density.c/87
  96 2984 0      DataNodeList     58.1818   27074.2763  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 1
  26 2979 0          Nextnode    222.3025   26581.4981  force_treeallocate()/forcetree.c/4820
 126 2990 0             DM_Vz    126.7529   27723.1856  subfind_density()/subfind_density.c/97
 101 2982 0         Dist2list    126.8688   26993.2857  subfind_density()/subfind_density.c/79
 101 2983 0    DataIndexTable     21.8182   27015.1039  subfind_density()/subfind_density.c/87
 101 2984 0      DataNodeList     58.1818   27073.2857  subfind_density()/subfind_density.c/89
 101 2985 0              Left    126.8688   27200.1544  subfind_density()/subfind_density.c/91
 101 2986 0             Right    126.8688   27327.0232  subfind_density()/subfind_density.c/92
  96 2985 0              Left    126.8614   27201.1377  subfind_density()/subfind_density.c/91
  26 2980 0            Father    221.9056   26803.4037  force_treeallocate()/forcetree.c/4827
  26 2981 0           Ngblist     63.4214   26866.8251  subfind_density()/subfind_density.c/78
  26 2982 0         Dist2list    126.8428   26993.6678  subfind_density()/subfind_density.c/79
 126 2991 0    	  DensDataGet    132.2567   27855.4422  subfind_density()/subfind_density.c/161
 101 2987 0              Todo     15.8586   27342.8818  subfind_density()/subfind_density.c/93
 101 2988 0             DM_Vx    126.8688   27469.7505  subfind_density()/subfind_density.c/95
 101 2989 0             DM_Vy    126.8688   27596.6193  subfind_density()/subfind_density.c/96
  96 2986 0             Right    126.8614   27327.9991  subfind_density()/subfind_density.c/92
  26 2983 0    DataIndexTable     21.8182   27015.4860  subfind_density()/subfind_density.c/87
  26 2984 0      DataNodeList     58.1818   27073.6678  subfind_density()/subfind_density.c/89
  26 2985 0              Left    126.8428   27200.5106  subfind_density()/subfind_density.c/91
 126 2992 0  	  DensDataResult     99.1925   27954.6347  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 101 2990 0             DM_Vz    126.8688   27723.4880  subfind_density()/subfind_density.c/97
 101 2991 0    	  DensDataGet    111.0558   27834.5439  subfind_density()/subfind_density.c/161
 101 2992 0  	  DensDataResult     83.2919   27917.8358  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  96 2987 0              Todo     15.8577   27343.8568  subfind_density()/subfind_density.c/93
  26 2986 0             Right    126.8428   27327.3534  subfind_density()/subfind_density.c/92
  26 2987 0              Todo     15.8554   27343.2087  subfind_density()/subfind_density.c/93
  26 2988 0             DM_Vx    126.8428   27470.0515  subfind_density()/subfind_density.c/95

Task=126: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.3653 MB).

Task=101: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.1642 MB).
task 101: endrun called with an error level of 812


  96 2988 0             DM_Vx    126.8614   27470.7182  subfind_density()/subfind_density.c/95
  26 2989 0             DM_Vy    126.8428   27596.8943  subfind_density()/subfind_density.c/96
  26 2990 0             DM_Vz    126.8428   27723.7371  subfind_density()/subfind_density.c/97
task 126: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 101
  96 2989 0             DM_Vy    126.8614   27597.5796  subfind_density()/subfind_density.c/96
  26 2991 0    	  DensDataGet    108.6965   27832.4336  subfind_density()/subfind_density.c/161
  26 2992 0  	  DensDataResult     81.5224   27913.9559  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 126
  96 2990 0             DM_Vz    126.8614   27724.4410  subfind_density()/subfind_density.c/97

Task=26: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.0441 MB).
task 26: endrun called with an error level of 812


  96 2991 0    	  DensDataGet    134.1343   27858.5753  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 26
  96 2992 0  	  DensDataResult    100.6007   27959.1760  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=96: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=40.824 MB).
task 96: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 96
  25 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  25 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  25 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  25 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  25 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  25 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  25 2973 0         FOF_PList    380.3735   18861.1927  fof_fof()/fof.c/249
  25 2974 0         FOF_GList    633.9558   19495.1485  fof_fof()/fof.c/337
  25 2975 0             Group     12.5529   19507.7013  fof_fof()/fof.c/380
  25 2976 0   DomainNodeIndex      0.3473   19508.0486  force_treeallocate()/forcetree.c/4800
 102 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 102 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 102 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 102 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 102 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
task 96: endrun called with an error level of 124


  25 2977 0        Nodes_base   3737.5399   23245.5885  force_treeallocate()/forcetree.c/4803
  25 2978 0     Extnodes_base   3114.6165   26360.2050  force_treeallocate()/forcetree.c/4811
  25 2979 0          Nextnode    222.3025   26582.5075  force_treeallocate()/forcetree.c/4820
  25 2980 0            Father    221.9056   26804.4131  force_treeallocate()/forcetree.c/4827
 102 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 102 2973 0         FOF_PList    380.1649   18860.9842  fof_fof()/fof.c/249
 102 2974 0         FOF_GList    633.6082   19494.5924  fof_fof()/fof.c/337
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 96:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 96
  25 2981 0           Ngblist     63.3956   26867.8087  subfind_density()/subfind_density.c/78
  25 2982 0         Dist2list    126.7912   26994.5999  subfind_density()/subfind_density.c/79
 102 2975 0             Group     11.0608   19505.6532  fof_fof()/fof.c/380
 102 2976 0   DomainNodeIndex      0.3473   19506.0005  force_treeallocate()/forcetree.c/4800
 102 2977 0        Nodes_base   3737.5399   23243.5403  force_treeallocate()/forcetree.c/4803
  25 2983 0    DataIndexTable     21.8182   27016.4180  subfind_density()/subfind_density.c/87
  25 2984 0      DataNodeList     58.1818   27074.5998  subfind_density()/subfind_density.c/89
  25 2985 0              Left    126.7912   27201.3910  subfind_density()/subfind_density.c/91
 102 2978 0     Extnodes_base   3114.6165   26358.1569  force_treeallocate()/forcetree.c/4811
 102 2979 0          Nextnode    222.3025   26580.4594  force_treeallocate()/forcetree.c/4820
 102 2980 0            Father    221.9056   26802.3650  force_treeallocate()/forcetree.c/4827
  25 2986 0             Right    126.7912   27328.1821  subfind_density()/subfind_density.c/92
  25 2987 0              Todo     15.8489   27344.0310  subfind_density()/subfind_density.c/93
 102 2981 0           Ngblist     63.3608   26865.7258  subfind_density()/subfind_density.c/78
 102 2982 0         Dist2list    126.7216   26992.4474  subfind_density()/subfind_density.c/79
 102 2983 0    DataIndexTable     21.8182   27014.2656  subfind_density()/subfind_density.c/87
  25 2988 0             DM_Vx    126.7912   27470.8222  subfind_density()/subfind_density.c/95
  25 2989 0             DM_Vy    126.7912   27597.6133  subfind_density()/subfind_density.c/96
 102 2984 0      DataNodeList     58.1818   27072.4474  subfind_density()/subfind_density.c/89
 102 2985 0              Left    126.7216   27199.1690  subfind_density()/subfind_density.c/91
  25 2990 0             DM_Vz    126.7912   27724.4045  subfind_density()/subfind_density.c/97
 102 2986 0             Right    126.7216   27325.8907  subfind_density()/subfind_density.c/92
  25 2991 0    	  DensDataGet    114.2482   27838.6527  subfind_density()/subfind_density.c/161
 102 2987 0              Todo     15.8402   27341.7309  subfind_density()/subfind_density.c/93
  25 2992 0  	  DensDataResult     85.6861   27924.3388  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 102 2988 0             DM_Vx    126.7216   27468.4525  subfind_density()/subfind_density.c/95

Task=25: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.6612 MB).
 102 2989 0             DM_Vy    126.7216   27595.1742  subfind_density()/subfind_density.c/96
task 25: endrun called with an error level of 812


 102 2990 0             DM_Vz    126.7216   27721.8958  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 25
 102 2991 0    	  DensDataGet    126.0955   27847.9913  subfind_density()/subfind_density.c/161
 102 2992 0  	  DensDataResult     94.5716   27942.5630  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=102: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.437 MB).
task 102: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 102
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  27 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  27 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  27 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  27 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  27 2973 0         FOF_PList    380.5651   18861.3844  fof_fof()/fof.c/249
  27 2974 0         FOF_GList    634.2752   19495.6596  fof_fof()/fof.c/337
  27 2975 0             Group     10.2587   19505.9183  fof_fof()/fof.c/380
  27 2976 0   DomainNodeIndex      0.3473   19506.2656  force_treeallocate()/forcetree.c/4800
  27 2977 0        Nodes_base   3737.5399   23243.8055  force_treeallocate()/forcetree.c/4803
  27 2978 0     Extnodes_base   3114.6165   26358.4220  force_treeallocate()/forcetree.c/4811
 100 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 100 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 100 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 100 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 100 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 100 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  27 2979 0          Nextnode    222.3025   26580.7245  force_treeallocate()/forcetree.c/4820
  27 2980 0            Father    221.9056   26802.6301  force_treeallocate()/forcetree.c/4827
  27 2981 0           Ngblist     63.4275   26866.0576  subfind_density()/subfind_density.c/78
  27 2982 0         Dist2list    126.8550   26992.9127  subfind_density()/subfind_density.c/79
 100 2973 0         FOF_PList    380.4676   18861.2869  fof_fof()/fof.c/249
 100 2974 0         FOF_GList    634.1127   19495.3996  fof_fof()/fof.c/337
 100 2975 0             Group     10.4963   19505.8960  fof_fof()/fof.c/380
  27 2983 0    DataIndexTable     21.8182   27014.7308  subfind_density()/subfind_density.c/87
  27 2984 0      DataNodeList     58.1818   27072.9126  subfind_density()/subfind_density.c/89
  27 2985 0              Left    126.8550   27199.7677  subfind_density()/subfind_density.c/91
  27 2986 0             Right    126.8550   27326.6227  subfind_density()/subfind_density.c/92
 100 2976 0   DomainNodeIndex      0.3473   19506.2433  force_treeallocate()/forcetree.c/4800
 100 2977 0        Nodes_base   3737.5399   23243.7831  force_treeallocate()/forcetree.c/4803
  27 2987 0              Todo     15.8569   27342.4796  subfind_density()/subfind_density.c/93
  27 2988 0             DM_Vx    126.8550   27469.3346  subfind_density()/subfind_density.c/95
 100 2978 0     Extnodes_base   3114.6165   26358.3997  force_treeallocate()/forcetree.c/4811
 100 2979 0          Nextnode    222.3025   26580.7022  force_treeallocate()/forcetree.c/4820
 100 2980 0            Father    221.9056   26802.6078  force_treeallocate()/forcetree.c/4827
  27 2989 0             DM_Vy    126.8550   27596.1897  subfind_density()/subfind_density.c/96
  27 2990 0             DM_Vz    126.8550   27723.0447  subfind_density()/subfind_density.c/97
  27 2991 0    	  DensDataGet    116.3664   27839.4111  subfind_density()/subfind_density.c/161
 100 2981 0           Ngblist     63.4113   26866.0190  subfind_density()/subfind_density.c/78
 100 2982 0         Dist2list    126.8225   26992.8416  subfind_density()/subfind_density.c/79
 100 2983 0    DataIndexTable     21.8182   27014.6598  subfind_density()/subfind_density.c/87
  27 2992 0  	  DensDataResult     87.2748   27926.6859  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=27: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.3141 MB).
task 27: endrun called with an error level of 812


 100 2984 0      DataNodeList     58.1818   27072.8416  subfind_density()/subfind_density.c/89
 100 2985 0              Left    126.8225   27199.6641  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 27
 100 2986 0             Right    126.8225   27326.4867  subfind_density()/subfind_density.c/92
 100 2987 0              Todo     15.8528   27342.3395  subfind_density()/subfind_density.c/93
 100 2988 0             DM_Vx    126.8225   27469.1620  subfind_density()/subfind_density.c/95
 100 2989 0             DM_Vy    126.8225   27595.9846  subfind_density()/subfind_density.c/96
 100 2990 0             DM_Vz    126.8225   27722.8071  subfind_density()/subfind_density.c/97
 100 2991 0    	  DensDataGet    111.3146   27834.1217  subfind_density()/subfind_density.c/161
 100 2992 0  	  DensDataResult     83.4859   27917.6076  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=100: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.3924 MB).
task 100: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 100
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  14 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  14 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  14 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  14 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  14 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  14 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 113 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 113 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 113 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 113 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 113 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 113 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 113 2973 0         FOF_PList    380.5321   18861.3513  fof_fof()/fof.c/249
 113 2974 0         FOF_GList    634.2202   19495.5715  fof_fof()/fof.c/337
 113 2975 0             Group     11.2382   19506.8097  fof_fof()/fof.c/380
 113 2976 0   DomainNodeIndex      0.3473   19507.1570  force_treeallocate()/forcetree.c/4800
 113 2977 0        Nodes_base   3737.5399   23244.6969  force_treeallocate()/forcetree.c/4803
 113 2978 0     Extnodes_base   3114.6165   26359.3134  force_treeallocate()/forcetree.c/4811
 113 2979 0          Nextnode    222.3025   26581.6159  force_treeallocate()/forcetree.c/4820
 113 2980 0            Father    221.9056   26803.5215  force_treeallocate()/forcetree.c/4827
 113 2981 0           Ngblist     63.4220   26866.9435  subfind_density()/subfind_density.c/78
 113 2982 0         Dist2list    126.8440   26993.7876  subfind_density()/subfind_density.c/79
 113 2983 0    DataIndexTable     21.8182   27015.6058  subfind_density()/subfind_density.c/87
 113 2984 0      DataNodeList     58.1818   27073.7875  subfind_density()/subfind_density.c/89
 113 2985 0              Left    126.8440   27200.6316  subfind_density()/subfind_density.c/91
  14 2973 0         FOF_PList    380.5594   18861.3786  fof_fof()/fof.c/249
  14 2974 0         FOF_GList    634.2657   19495.6443  fof_fof()/fof.c/337
  14 2975 0             Group     10.2587   19505.9030  fof_fof()/fof.c/380
 113 2986 0             Right    126.8440   27327.4756  subfind_density()/subfind_density.c/92
 113 2987 0              Todo     15.8555   27343.3311  subfind_density()/subfind_density.c/93
 113 2988 0             DM_Vx    126.8440   27470.1751  subfind_density()/subfind_density.c/95
 113 2989 0             DM_Vy    126.8440   27597.0192  subfind_density()/subfind_density.c/96
 113 2990 0             DM_Vz    126.8440   27723.8632  subfind_density()/subfind_density.c/97
 113 2991 0    	  DensDataGet    117.8407   27841.7039  subfind_density()/subfind_density.c/161
  14 2976 0   DomainNodeIndex      0.3473   19506.2504  force_treeallocate()/forcetree.c/4800
  14 2977 0        Nodes_base   3737.5399   23243.7902  force_treeallocate()/forcetree.c/4803
  14 2978 0     Extnodes_base   3114.6165   26358.4068  force_treeallocate()/forcetree.c/4811
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 113 2992 0  	  DensDataResult     88.3805   27930.0844  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=113: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.9156 MB).
task 113: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  14 2979 0          Nextnode    222.3025   26580.7093  force_treeallocate()/forcetree.c/4820
  14 2980 0            Father    221.9056   26802.6148  force_treeallocate()/forcetree.c/4827
  14 2981 0           Ngblist     63.4266   26866.0414  subfind_density()/subfind_density.c/78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   2 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   2 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   2 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   2 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   2 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   2 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
   2 2973 0         FOF_PList    380.5354   18861.3546  fof_fof()/fof.c/249
   2 2974 0         FOF_GList    634.2256   19495.5802  fof_fof()/fof.c/337
   2 2975 0             Group     11.0058   19506.5860  fof_fof()/fof.c/380
   2 2976 0   DomainNodeIndex      0.3473   19506.9333  force_treeallocate()/forcetree.c/4800
   2 2977 0        Nodes_base   3737.5399   23244.4732  force_treeallocate()/forcetree.c/4803
   2 2978 0     Extnodes_base   3114.6165   26359.0897  force_treeallocate()/forcetree.c/4811
 119 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 119 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 119 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 119 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 119 2973 0         FOF_PList    380.5323   18861.3515  fof_fof()/fof.c/249
 119 2974 0         FOF_GList    634.2205   19495.5720  fof_fof()/fof.c/337
 119 2975 0             Group     11.5163   19507.0883  fof_fof()/fof.c/380
 119 2976 0   DomainNodeIndex      0.3473   19507.4356  force_treeallocate()/forcetree.c/4800
 119 2977 0        Nodes_base   3737.5399   23244.9754  force_treeallocate()/forcetree.c/4803
 119 2978 0     Extnodes_base   3114.6165   26359.5920  force_treeallocate()/forcetree.c/4811
 119 2979 0          Nextnode    222.3025   26581.8945  force_treeallocate()/forcetree.c/4820
 119 2980 0            Father    221.9056   26803.8001  force_treeallocate()/forcetree.c/4827
 114 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 114 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 114 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 114 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 114 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 114 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 114 2973 0         FOF_PList    380.5198   18861.3391  fof_fof()/fof.c/249
 114 2974 0         FOF_GList    634.1997   19495.5388  fof_fof()/fof.c/337
 114 2975 0             Group     10.2587   19505.7975  fof_fof()/fof.c/380
 117 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 117 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 117 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 117 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 117 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 117 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 117 2973 0         FOF_PList    380.4389   18861.2582  fof_fof()/fof.c/249
 117 2974 0         FOF_GList    634.0649   19495.3230  fof_fof()/fof.c/337
 117 2975 0             Group     11.3658   19506.6889  fof_fof()/fof.c/380
 117 2976 0   DomainNodeIndex      0.3473   19507.0362  force_treeallocate()/forcetree.c/4800
 117 2977 0        Nodes_base   3737.5399   23244.5760  force_treeallocate()/forcetree.c/4803
 117 2978 0     Extnodes_base   3114.6165   26359.1926  force_treeallocate()/forcetree.c/4811
 117 2979 0          Nextnode    222.3025   26581.4951  force_treeallocate()/forcetree.c/4820
 117 2980 0            Father    221.9056   26803.4007  force_treeallocate()/forcetree.c/4827
 117 2981 0           Ngblist     63.4065   26866.8072  subfind_density()/subfind_density.c/78
  10 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  10 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  10 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  10 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  10 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  10 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  10 2973 0         FOF_PList    380.5309   18861.3501  fof_fof()/fof.c/249
  10 2974 0         FOF_GList    634.2181   19495.5682  fof_fof()/fof.c/337
  10 2975 0             Group     11.5671   19507.1353  fof_fof()/fof.c/380
  10 2976 0   DomainNodeIndex      0.3473   19507.4827  force_treeallocate()/forcetree.c/4800
 108 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 108 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 108 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 108 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 108 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 108 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 108 2973 0         FOF_PList    380.5955   18861.4147  fof_fof()/fof.c/249
 108 2974 0         FOF_GList    634.3258   19495.7404  fof_fof()/fof.c/337
 108 2975 0             Group     11.3845   19507.1250  fof_fof()/fof.c/380
 108 2976 0   DomainNodeIndex      0.3473   19507.4723  force_treeallocate()/forcetree.c/4800
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 113
  13 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  13 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  13 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  13 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  13 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  13 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  13 2973 0         FOF_PList    380.2992   18861.1185  fof_fof()/fof.c/249
  13 2974 0         FOF_GList    633.8321   19494.9505  fof_fof()/fof.c/337
  13 2975 0             Group     11.4779   19506.4284  fof_fof()/fof.c/380
  13 2976 0   DomainNodeIndex      0.3473   19506.7757  force_treeallocate()/forcetree.c/4800
  13 2977 0        Nodes_base   3737.5399   23244.3156  force_treeallocate()/forcetree.c/4803
  13 2978 0     Extnodes_base   3114.6165   26358.9321  force_treeallocate()/forcetree.c/4811
  13 2979 0          Nextnode    222.3025   26581.2346  force_treeallocate()/forcetree.c/4820
  13 2980 0            Father    221.9056   26803.1402  force_treeallocate()/forcetree.c/4827
 121 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 121 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 121 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 121 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 121 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 121 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 121 2973 0         FOF_PList    380.6006   18861.4198  fof_fof()/fof.c/249
 121 2974 0         FOF_GList    634.3343   19495.7541  fof_fof()/fof.c/337
 121 2975 0             Group     11.4561   19507.2102  fof_fof()/fof.c/380
  99 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  99 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  99 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  99 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  99 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  99 2973 0         FOF_PList    380.5354   18861.3547  fof_fof()/fof.c/249
  99 2974 0         FOF_GList    634.2257   19495.5804  fof_fof()/fof.c/337
  99 2975 0             Group     10.2816   19505.8620  fof_fof()/fof.c/380
  99 2976 0   DomainNodeIndex      0.3473   19506.2093  force_treeallocate()/forcetree.c/4800
  99 2977 0        Nodes_base   3737.5399   23243.7491  force_treeallocate()/forcetree.c/4803
  99 2978 0     Extnodes_base   3114.6165   26358.3657  force_treeallocate()/forcetree.c/4811
  99 2979 0          Nextnode    222.3025   26580.6682  force_treeallocate()/forcetree.c/4820
  99 2980 0            Father    221.9056   26802.5738  force_treeallocate()/forcetree.c/4827
  28 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  28 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  28 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  28 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  28 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  28 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  28 2973 0         FOF_PList    380.4915   18861.3108  fof_fof()/fof.c/249
  28 2974 0         FOF_GList    634.1525   19495.4633  fof_fof()/fof.c/337
  28 2975 0             Group     12.1129   19507.5762  fof_fof()/fof.c/380
  28 2976 0   DomainNodeIndex      0.3473   19507.9235  force_treeallocate()/forcetree.c/4800
  28 2977 0        Nodes_base   3737.5399   23245.4634  force_treeallocate()/forcetree.c/4803
  19 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  19 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  19 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  19 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  19 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  19 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  19 2973 0         FOF_PList    380.4186   18861.2378  fof_fof()/fof.c/249
  19 2974 0         FOF_GList    634.0310   19495.2688  fof_fof()/fof.c/337
  19 2975 0             Group     12.1752   19507.4440  fof_fof()/fof.c/380
  19 2976 0   DomainNodeIndex      0.3473   19507.7913  force_treeallocate()/forcetree.c/4800
  19 2977 0        Nodes_base   3737.5399   23245.3312  force_treeallocate()/forcetree.c/4803
  19 2978 0     Extnodes_base   3114.6165   26359.9477  force_treeallocate()/forcetree.c/4811
  19 2979 0          Nextnode    222.3025   26582.2502  force_treeallocate()/forcetree.c/4820
  19 2980 0            Father    221.9056   26804.1558  force_treeallocate()/forcetree.c/4827
  19 2981 0           Ngblist     63.4031   26867.5589  subfind_density()/subfind_density.c/78
  19 2982 0         Dist2list    126.8062   26994.3651  subfind_density()/subfind_density.c/79
  19 2983 0    DataIndexTable     21.8182   27016.1833  subfind_density()/subfind_density.c/87
  21 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  21 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  21 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  21 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  21 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  21 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  21 2973 0         FOF_PList    380.6038   18861.4230  fof_fof()/fof.c/249
  21 2974 0         FOF_GList    634.3397   19495.7627  fof_fof()/fof.c/337
  21 2975 0             Group     10.8398   19506.6025  fof_fof()/fof.c/380
  21 2976 0   DomainNodeIndex      0.3473   19506.9498  force_treeallocate()/forcetree.c/4800
  21 2977 0        Nodes_base   3737.5399   23244.4897  force_treeallocate()/forcetree.c/4803
  21 2978 0     Extnodes_base   3114.6165   26359.1062  force_treeallocate()/forcetree.c/4811
  21 2979 0          Nextnode    222.3025   26581.4087  force_treeallocate()/forcetree.c/4820
  21 2980 0            Father    221.9056   26803.3143  force_treeallocate()/forcetree.c/4827
 110 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 110 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 110 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 110 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 110 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 110 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 110 2973 0         FOF_PList    380.5141   18861.3333  fof_fof()/fof.c/249
 110 2974 0         FOF_GList    634.1901   19495.5235  fof_fof()/fof.c/337
 110 2975 0             Group     12.7064   19508.2299  fof_fof()/fof.c/380
   5 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   5 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   5 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   5 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   5 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   5 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
   5 2973 0         FOF_PList    380.5505   18861.3698  fof_fof()/fof.c/249
   5 2974 0         FOF_GList    634.2509   19495.6206  fof_fof()/fof.c/337
   5 2975 0             Group     10.8854   19506.5061  fof_fof()/fof.c/380
   5 2976 0   DomainNodeIndex      0.3473   19506.8534  force_treeallocate()/forcetree.c/4800
   5 2977 0        Nodes_base   3737.5399   23244.3932  force_treeallocate()/forcetree.c/4803
   5 2978 0     Extnodes_base   3114.6165   26359.0098  force_treeallocate()/forcetree.c/4811
   5 2979 0          Nextnode    222.3025   26581.3123  force_treeallocate()/forcetree.c/4820
   5 2980 0            Father    221.9056   26803.2179  force_treeallocate()/forcetree.c/4827
   5 2981 0           Ngblist     63.4251   26866.6430  subfind_density()/subfind_density.c/78
   5 2982 0         Dist2list    126.8502   26993.4931  subfind_density()/subfind_density.c/79
   5 2983 0    DataIndexTable     21.8182   27015.3113  subfind_density()/subfind_density.c/87
   5 2984 0      DataNodeList     58.1818   27073.4931  subfind_density()/subfind_density.c/89
   5 2985 0              Left    126.8502   27200.3433  subfind_density()/subfind_density.c/91
   5 2986 0             Right    126.8502   27327.1935  subfind_density()/subfind_density.c/92
   5 2987 0              Todo     15.8563   27343.0497  subfind_density()/subfind_density.c/93
   5 2988 0             DM_Vx    126.8502   27469.8999  subfind_density()/subfind_density.c/95
   5 2989 0             DM_Vy    126.8502   27596.7501  subfind_density()/subfind_density.c/96
   5 2990 0             DM_Vz    126.8502   27723.6003  subfind_density()/subfind_density.c/97
   5 2991 0    	  DensDataGet    124.6728   27848.2730  subfind_density()/subfind_density.c/161
  14 2982 0         Dist2list    126.8531   26992.8945  subfind_density()/subfind_density.c/79
  14 2983 0    DataIndexTable     21.8182   27014.7127  subfind_density()/subfind_density.c/87
 106 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 106 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 106 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 106 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 106 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 106 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 106 2973 0         FOF_PList    380.5944   18861.4136  fof_fof()/fof.c/249
 106 2974 0         FOF_GList    634.3240   19495.7376  fof_fof()/fof.c/337
 106 2975 0             Group     12.2032   19507.9408  fof_fof()/fof.c/380
 106 2976 0   DomainNodeIndex      0.3473   19508.2881  force_treeallocate()/forcetree.c/4800
 106 2977 0        Nodes_base   3737.5399   23245.8280  force_treeallocate()/forcetree.c/4803
 106 2978 0     Extnodes_base   3114.6165   26360.4445  force_treeallocate()/forcetree.c/4811
  17 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  17 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  17 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  17 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  17 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  17 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  17 2973 0         FOF_PList    380.0871   18860.9064  fof_fof()/fof.c/249
  17 2974 0         FOF_GList    633.4785   19494.3849  fof_fof()/fof.c/337
  17 2975 0             Group     12.4698   19506.8548  fof_fof()/fof.c/380
  17 2976 0   DomainNodeIndex      0.3473   19507.2021  force_treeallocate()/forcetree.c/4800
  17 2977 0        Nodes_base   3737.5399   23244.7419  force_treeallocate()/forcetree.c/4803
  17 2978 0     Extnodes_base   3114.6165   26359.3585  force_treeallocate()/forcetree.c/4811
  17 2979 0          Nextnode    222.3025   26581.6610  force_treeallocate()/forcetree.c/4820
  17 2980 0            Father    221.9056   26803.5666  force_treeallocate()/forcetree.c/4827
  17 2981 0           Ngblist     63.3479   26866.9144  subfind_density()/subfind_density.c/78
  17 2982 0         Dist2list    126.6957   26993.6101  subfind_density()/subfind_density.c/79
   2 2979 0          Nextnode    222.3025   26581.3922  force_treeallocate()/forcetree.c/4820
   2 2980 0            Father    221.9056   26803.2978  force_treeallocate()/forcetree.c/4827
 119 2981 0           Ngblist     63.4221   26867.2221  subfind_density()/subfind_density.c/78
 119 2982 0         Dist2list    126.8441   26994.0662  subfind_density()/subfind_density.c/79
 119 2983 0    DataIndexTable     21.8182   27015.8844  subfind_density()/subfind_density.c/87
 119 2984 0      DataNodeList     58.1818   27074.0662  subfind_density()/subfind_density.c/89
 119 2985 0              Left    126.8441   27200.9103  subfind_density()/subfind_density.c/91
 114 2976 0   DomainNodeIndex      0.3473   19506.1448  force_treeallocate()/forcetree.c/4800
 114 2977 0        Nodes_base   3737.5399   23243.6847  force_treeallocate()/forcetree.c/4803
 117 2982 0         Dist2list    126.8130   26993.6201  subfind_density()/subfind_density.c/79
 117 2983 0    DataIndexTable     21.8182   27015.4383  subfind_density()/subfind_density.c/87
 117 2984 0      DataNodeList     58.1818   27073.6201  subfind_density()/subfind_density.c/89
 117 2985 0              Left    126.8130   27200.4331  subfind_density()/subfind_density.c/91
 117 2986 0             Right    126.8130   27327.2460  subfind_density()/subfind_density.c/92
  10 2977 0        Nodes_base   3737.5399   23245.0225  force_treeallocate()/forcetree.c/4803
  10 2978 0     Extnodes_base   3114.6165   26359.6391  force_treeallocate()/forcetree.c/4811
  10 2979 0          Nextnode    222.3025   26581.9416  force_treeallocate()/forcetree.c/4820
 108 2977 0        Nodes_base   3737.5399   23245.0121  force_treeallocate()/forcetree.c/4803
 108 2978 0     Extnodes_base   3114.6165   26359.6287  force_treeallocate()/forcetree.c/4811
 108 2979 0          Nextnode    222.3025   26581.9312  force_treeallocate()/forcetree.c/4820
 108 2980 0            Father    221.9056   26803.8368  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  13 2981 0           Ngblist     63.3832   26866.5234  subfind_density()/subfind_density.c/78
  13 2982 0         Dist2list    126.7664   26993.2899  subfind_density()/subfind_density.c/79
  13 2983 0    DataIndexTable     21.8182   27015.1080  subfind_density()/subfind_density.c/87
  13 2984 0      DataNodeList     58.1818   27073.2898  subfind_density()/subfind_density.c/89
  13 2985 0              Left    126.7664   27200.0562  subfind_density()/subfind_density.c/91
 121 2976 0   DomainNodeIndex      0.3473   19507.5575  force_treeallocate()/forcetree.c/4800
 121 2977 0        Nodes_base   3737.5399   23245.0973  force_treeallocate()/forcetree.c/4803
 121 2978 0     Extnodes_base   3114.6165   26359.7139  force_treeallocate()/forcetree.c/4811
 121 2979 0          Nextnode    222.3025   26582.0164  force_treeallocate()/forcetree.c/4820
 121 2980 0            Father    221.9056   26803.9220  force_treeallocate()/forcetree.c/4827
  99 2981 0           Ngblist     63.4226   26865.9963  subfind_density()/subfind_density.c/78
  99 2982 0         Dist2list    126.8451   26992.8415  subfind_density()/subfind_density.c/79
  99 2983 0    DataIndexTable     21.8182   27014.6597  subfind_density()/subfind_density.c/87
  99 2984 0      DataNodeList     58.1818   27072.8415  subfind_density()/subfind_density.c/89
  28 2978 0     Extnodes_base   3114.6165   26360.0799  force_treeallocate()/forcetree.c/4811
  19 2984 0      DataNodeList     58.1818   27074.3651  subfind_density()/subfind_density.c/89
  19 2985 0              Left    126.8062   27201.1713  subfind_density()/subfind_density.c/91
  19 2986 0             Right    126.8062   27327.9775  subfind_density()/subfind_density.c/92
  19 2987 0              Todo     15.8508   27343.8282  subfind_density()/subfind_density.c/93
  19 2988 0             DM_Vx    126.8062   27470.6344  subfind_density()/subfind_density.c/95
  19 2989 0             DM_Vy    126.8062   27597.4406  subfind_density()/subfind_density.c/96
  21 2981 0           Ngblist     63.4340   26866.7483  subfind_density()/subfind_density.c/78
  21 2982 0         Dist2list    126.8679   26993.6162  subfind_density()/subfind_density.c/79
  21 2983 0    DataIndexTable     21.8182   27015.4344  subfind_density()/subfind_density.c/87
  21 2984 0      DataNodeList     58.1818   27073.6162  subfind_density()/subfind_density.c/89
 110 2976 0   DomainNodeIndex      0.3473   19508.5772  force_treeallocate()/forcetree.c/4800
 110 2977 0        Nodes_base   3737.5399   23246.1171  force_treeallocate()/forcetree.c/4803
 110 2978 0     Extnodes_base   3114.6165   26360.7336  force_treeallocate()/forcetree.c/4811
 110 2979 0          Nextnode    222.3025   26583.0361  force_treeallocate()/forcetree.c/4820
   5 2992 0  	  DensDataResult     93.5046   27941.7776  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=5: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.2224 MB).
task 5: endrun called with an error level of 812


  14 2984 0      DataNodeList     58.1818   27072.8945  subfind_density()/subfind_density.c/89
 106 2979 0          Nextnode    222.3025   26582.7470  force_treeallocate()/forcetree.c/4820
 106 2980 0            Father    221.9056   26804.6526  force_treeallocate()/forcetree.c/4827
 106 2981 0           Ngblist     63.4324   26868.0850  subfind_density()/subfind_density.c/78
  17 2983 0    DataIndexTable     21.8182   27015.4283  subfind_density()/subfind_density.c/87
  17 2984 0      DataNodeList     58.1818   27073.6101  subfind_density()/subfind_density.c/89
  17 2985 0              Left    126.6957   27200.3058  subfind_density()/subfind_density.c/91
  17 2986 0             Right    126.6957   27327.0015  subfind_density()/subfind_density.c/92
  17 2987 0              Todo     15.8370   27342.8385  subfind_density()/subfind_density.c/93
   2 2981 0           Ngblist     63.4226   26866.7204  subfind_density()/subfind_density.c/78
   2 2982 0         Dist2list    126.8451   26993.5655  subfind_density()/subfind_density.c/79
   2 2983 0    DataIndexTable     21.8182   27015.3837  subfind_density()/subfind_density.c/87
 119 2986 0             Right    126.8441   27327.7544  subfind_density()/subfind_density.c/92
 119 2987 0              Todo     15.8555   27343.6099  subfind_density()/subfind_density.c/93
 119 2988 0             DM_Vx    126.8441   27470.4540  subfind_density()/subfind_density.c/95
 114 2978 0     Extnodes_base   3114.6165   26358.3012  force_treeallocate()/forcetree.c/4811
 114 2979 0          Nextnode    222.3025   26580.6037  force_treeallocate()/forcetree.c/4820
 114 2980 0            Father    221.9056   26802.5093  force_treeallocate()/forcetree.c/4827
 117 2987 0              Todo     15.8516   27343.0977  subfind_density()/subfind_density.c/93
 117 2988 0             DM_Vx    126.8130   27469.9106  subfind_density()/subfind_density.c/95
 117 2989 0             DM_Vy    126.8130   27596.7236  subfind_density()/subfind_density.c/96
 117 2990 0             DM_Vz    126.8130   27723.5366  subfind_density()/subfind_density.c/97
  10 2980 0            Father    221.9056   26803.8471  force_treeallocate()/forcetree.c/4827
  10 2981 0           Ngblist     63.4218   26867.2690  subfind_density()/subfind_density.c/78
 108 2981 0           Ngblist     63.4326   26867.2693  subfind_density()/subfind_density.c/78
 108 2982 0         Dist2list    126.8652   26994.1345  subfind_density()/subfind_density.c/79
 108 2983 0    DataIndexTable     21.8182   27015.9527  subfind_density()/subfind_density.c/87
 112 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 112 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 112 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 112 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 112 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 112 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  13 2986 0             Right    126.7664   27326.8226  subfind_density()/subfind_density.c/92
  13 2987 0              Todo     15.8458   27342.6684  subfind_density()/subfind_density.c/93
  13 2988 0             DM_Vx    126.7664   27469.4349  subfind_density()/subfind_density.c/95
 121 2981 0           Ngblist     63.4334   26867.3554  subfind_density()/subfind_density.c/78
 121 2982 0         Dist2list    126.8669   26994.2223  subfind_density()/subfind_density.c/79
  99 2985 0              Left    126.8451   27199.6866  subfind_density()/subfind_density.c/91
  99 2986 0             Right    126.8451   27326.5318  subfind_density()/subfind_density.c/92
  99 2987 0              Todo     15.8556   27342.3874  subfind_density()/subfind_density.c/93
  99 2988 0             DM_Vx    126.8451   27469.2325  subfind_density()/subfind_density.c/95
  28 2979 0          Nextnode    222.3025   26582.3824  force_treeallocate()/forcetree.c/4820
  28 2980 0            Father    221.9056   26804.2880  force_treeallocate()/forcetree.c/4827
  28 2981 0           Ngblist     63.4153   26867.7033  subfind_density()/subfind_density.c/78
  19 2990 0             DM_Vz    126.8062   27724.2468  subfind_density()/subfind_density.c/97
  19 2991 0    	  DensDataGet    123.6570   27847.9039  subfind_density()/subfind_density.c/161
  19 2992 0  	  DensDataResult     92.7428   27940.6467  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  21 2985 0              Left    126.8679   27200.4841  subfind_density()/subfind_density.c/91
  21 2986 0             Right    126.8679   27327.3521  subfind_density()/subfind_density.c/92
  21 2987 0              Todo     15.8585   27343.2105  subfind_density()/subfind_density.c/93
 110 2980 0            Father    221.9056   26804.9417  force_treeallocate()/forcetree.c/4827
 110 2981 0           Ngblist     63.4190   26868.3607  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 5
  14 2985 0              Left    126.8531   27199.7477  subfind_density()/subfind_density.c/91
 106 2982 0         Dist2list    126.8648   26994.9498  subfind_density()/subfind_density.c/79
 106 2983 0    DataIndexTable     21.8182   27016.7680  subfind_density()/subfind_density.c/87
 106 2984 0      DataNodeList     58.1818   27074.9498  subfind_density()/subfind_density.c/89
  17 2988 0             DM_Vx    126.6957   27469.5342  subfind_density()/subfind_density.c/95
  17 2989 0             DM_Vy    126.6957   27596.2299  subfind_density()/subfind_density.c/96
  17 2990 0             DM_Vz    126.6957   27722.9256  subfind_density()/subfind_density.c/97
   2 2984 0      DataNodeList     58.1818   27073.5655  subfind_density()/subfind_density.c/89
   2 2985 0              Left    126.8451   27200.4106  subfind_density()/subfind_density.c/91
   2 2986 0             Right    126.8451   27327.2557  subfind_density()/subfind_density.c/92
 119 2989 0             DM_Vy    126.8441   27597.2981  subfind_density()/subfind_density.c/96
 119 2990 0             DM_Vz    126.8441   27724.1422  subfind_density()/subfind_density.c/97
 119 2991 0    	  DensDataGet    109.1656   27833.3078  subfind_density()/subfind_density.c/161
 114 2981 0           Ngblist     63.4200   26865.9293  subfind_density()/subfind_density.c/78
 114 2982 0         Dist2list    126.8399   26992.7692  subfind_density()/subfind_density.c/79
 117 2991 0    	  DensDataGet    112.5973   27836.1339  subfind_density()/subfind_density.c/161
 117 2992 0  	  DensDataResult     84.4480   27920.5818  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=117: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.4182 MB).
  10 2982 0         Dist2list    126.8436   26994.1126  subfind_density()/subfind_density.c/79
  10 2983 0    DataIndexTable     21.8182   27015.9308  subfind_density()/subfind_density.c/87
  10 2984 0      DataNodeList     58.1818   27074.1125  subfind_density()/subfind_density.c/89
  10 2985 0              Left    126.8436   27200.9562  subfind_density()/subfind_density.c/91
 108 2984 0      DataNodeList     58.1818   27074.1345  subfind_density()/subfind_density.c/89
 108 2985 0              Left    126.8652   27200.9996  subfind_density()/subfind_density.c/91
 108 2986 0             Right    126.8652   27327.8648  subfind_density()/subfind_density.c/92
 112 2973 0         FOF_PList    380.5011   18861.3204  fof_fof()/fof.c/249
 112 2974 0         FOF_GList    634.1685   19495.4889  fof_fof()/fof.c/337
 112 2975 0             Group     10.7246   19506.2135  fof_fof()/fof.c/380
  13 2989 0             DM_Vy    126.7664   27596.2013  subfind_density()/subfind_density.c/96
  13 2990 0             DM_Vz    126.7664   27722.9677  subfind_density()/subfind_density.c/97
  13 2991 0    	  DensDataGet    117.7209   27840.6886  subfind_density()/subfind_density.c/161
 121 2983 0    DataIndexTable     21.8182   27016.0404  subfind_density()/subfind_density.c/87
 121 2984 0      DataNodeList     58.1818   27074.2222  subfind_density()/subfind_density.c/89
 121 2985 0              Left    126.8669   27201.0891  subfind_density()/subfind_density.c/91
  99 2989 0             DM_Vy    126.8451   27596.0777  subfind_density()/subfind_density.c/96
  99 2990 0             DM_Vz    126.8451   27722.9228  subfind_density()/subfind_density.c/97
  99 2991 0    	  DensDataGet    117.3898   27840.3127  subfind_density()/subfind_density.c/161
  28 2982 0         Dist2list    126.8305   26994.5338  subfind_density()/subfind_density.c/79
  28 2983 0    DataIndexTable     21.8182   27016.3519  subfind_density()/subfind_density.c/87

Task=19: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.3533 MB).
task 19: endrun called with an error level of 812


  21 2988 0             DM_Vx    126.8679   27470.0785  subfind_density()/subfind_density.c/95
  21 2989 0             DM_Vy    126.8679   27596.9464  subfind_density()/subfind_density.c/96
  21 2990 0             DM_Vz    126.8679   27723.8144  subfind_density()/subfind_density.c/97
 110 2982 0         Dist2list    126.8380   26995.1987  subfind_density()/subfind_density.c/79
 110 2983 0    DataIndexTable     21.8182   27017.0169  subfind_density()/subfind_density.c/87
 110 2984 0      DataNodeList     58.1818   27075.1987  subfind_density()/subfind_density.c/89
  14 2986 0             Right    126.8531   27326.6008  subfind_density()/subfind_density.c/92
 106 2985 0              Left    126.8648   27201.8146  subfind_density()/subfind_density.c/91
 106 2986 0             Right    126.8648   27328.6794  subfind_density()/subfind_density.c/92
 106 2987 0              Todo     15.8581   27344.5375  subfind_density()/subfind_density.c/93
  17 2991 0    	  DensDataGet    111.4220   27834.3476  subfind_density()/subfind_density.c/161
  17 2992 0  	  DensDataResult     83.5665   27917.9141  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   2 2987 0              Todo     15.8556   27343.1114  subfind_density()/subfind_density.c/93
   2 2988 0             DM_Vx    126.8451   27469.9565  subfind_density()/subfind_density.c/95
 119 2992 0  	  DensDataResult     81.8742   27915.1820  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=119: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.818 MB).
task 119: endrun called with an error level of 812


 114 2983 0    DataIndexTable     21.8182   27014.5874  subfind_density()/subfind_density.c/87
 114 2984 0      DataNodeList     58.1818   27072.7692  subfind_density()/subfind_density.c/89
 114 2985 0              Left    126.8399   27199.6091  subfind_density()/subfind_density.c/91
task 117: endrun called with an error level of 812


  10 2986 0             Right    126.8436   27327.7998  subfind_density()/subfind_density.c/92
  10 2987 0              Todo     15.8555   27343.6552  subfind_density()/subfind_density.c/93
  10 2988 0             DM_Vx    126.8436   27470.4989  subfind_density()/subfind_density.c/95
 108 2987 0              Todo     15.8581   27343.7229  subfind_density()/subfind_density.c/93
 108 2988 0             DM_Vx    126.8652   27470.5881  subfind_density()/subfind_density.c/95
 108 2989 0             DM_Vy    126.8652   27597.4532  subfind_density()/subfind_density.c/96
 112 2976 0   DomainNodeIndex      0.3473   19506.5608  force_treeallocate()/forcetree.c/4800
 112 2977 0        Nodes_base   3737.5399   23244.1006  force_treeallocate()/forcetree.c/4803
  13 2992 0  	  DensDataResult     88.2907   27928.9793  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=13: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.0207 MB).
task 13: endrun called with an error level of 812


 121 2986 0             Right    126.8669   27327.9559  subfind_density()/subfind_density.c/92
 121 2987 0              Todo     15.8584   27343.8143  subfind_density()/subfind_density.c/93
  99 2992 0  	  DensDataResult     88.0424   27928.3550  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=99: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.645 MB).
task 99: endrun called with an error level of 812


  28 2984 0      DataNodeList     58.1818   27074.5337  subfind_density()/subfind_density.c/89
  28 2985 0              Left    126.8305   27201.3642  subfind_density()/subfind_density.c/91
  28 2986 0             Right    126.8305   27328.1947  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 19
  21 2991 0    	  DensDataGet    137.8301   27861.6445  subfind_density()/subfind_density.c/161
  21 2992 0  	  DensDataResult    103.3726   27965.0171  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=21: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=34.9829 MB).
 110 2985 0              Left    126.8380   27202.0367  subfind_density()/subfind_density.c/91
 110 2986 0             Right    126.8380   27328.8748  subfind_density()/subfind_density.c/92
 110 2987 0              Todo     15.8548   27344.7295  subfind_density()/subfind_density.c/93
  14 2987 0              Todo     15.8566   27342.4574  subfind_density()/subfind_density.c/93
 106 2988 0             DM_Vx    126.8648   27471.4023  subfind_density()/subfind_density.c/95
 106 2989 0             DM_Vy    126.8648   27598.2671  subfind_density()/subfind_density.c/96
 106 2990 0             DM_Vz    126.8648   27725.1319  subfind_density()/subfind_density.c/97

Task=17: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.0859 MB).
task 17: endrun called with an error level of 812


   2 2989 0             DM_Vy    126.8451   27596.8016  subfind_density()/subfind_density.c/96
   2 2990 0             DM_Vz    126.8451   27723.6467  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 119
 114 2986 0             Right    126.8399   27326.4491  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 117
  10 2989 0             DM_Vy    126.8436   27597.3425  subfind_density()/subfind_density.c/96
  10 2990 0             DM_Vz    126.8436   27724.1861  subfind_density()/subfind_density.c/97
 108 2990 0             DM_Vz    126.8652   27724.3184  subfind_density()/subfind_density.c/97
 108 2991 0    	  DensDataGet    109.6928   27834.0112  subfind_density()/subfind_density.c/161
 112 2978 0     Extnodes_base   3114.6165   26358.7172  force_treeallocate()/forcetree.c/4811
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 13
 121 2988 0             DM_Vx    126.8669   27470.6811  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 99
  28 2987 0              Todo     15.8538   27344.0486  subfind_density()/subfind_density.c/93
task 21: endrun called with an error level of 812


 110 2988 0             DM_Vx    126.8380   27471.5675  subfind_density()/subfind_density.c/95
  14 2988 0             DM_Vx    126.8531   27469.3106  subfind_density()/subfind_density.c/95
 106 2991 0    	  DensDataGet    117.2620   27842.3939  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 17
   2 2991 0    	  DensDataGet    124.4307   27848.0775  subfind_density()/subfind_density.c/161
   2 2992 0  	  DensDataResult     93.3230   27941.4005  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=2: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.5995 MB).
task 2: endrun called with an error level of 812


 114 2987 0              Todo     15.8550   27342.3041  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  10 2991 0    	  DensDataGet    127.7097   27851.8958  subfind_density()/subfind_density.c/161
  10 2992 0  	  DensDataResult     95.7822   27947.6780  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 108 2992 0  	  DensDataResult     82.2696   27916.2808  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 112 2979 0          Nextnode    222.3025   26581.0197  force_treeallocate()/forcetree.c/4820
 121 2989 0             DM_Vy    126.8669   27597.5480  subfind_density()/subfind_density.c/96
  28 2988 0             DM_Vx    126.8305   27470.8791  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 21
 110 2989 0             DM_Vy    126.8380   27598.4056  subfind_density()/subfind_density.c/96
  14 2989 0             DM_Vy    126.8531   27596.1637  subfind_density()/subfind_density.c/96
 106 2992 0  	  DensDataResult     87.9465   27930.3404  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 2
 114 2988 0             DM_Vx    126.8399   27469.1440  subfind_density()/subfind_density.c/95
 116 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 116 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 116 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 116 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 116 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 116 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570

Task=10: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.322 MB).
task 10: endrun called with an error level of 812



Task=108: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.7192 MB).
task 108: endrun called with an error level of 812


 112 2980 0            Father    221.9056   26802.9253  force_treeallocate()/forcetree.c/4827
 121 2990 0             DM_Vz    126.8669   27724.4148  subfind_density()/subfind_density.c/97
  28 2989 0             DM_Vy    126.8305   27597.7096  subfind_density()/subfind_density.c/96
 110 2990 0             DM_Vz    126.8380   27725.2436  subfind_density()/subfind_density.c/97
  14 2990 0             DM_Vz    126.8531   27723.0168  subfind_density()/subfind_density.c/97

Task=106: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.6596 MB).
task 106: endrun called with an error level of 812


  16 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  16 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  16 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  16 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  16 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 114 2989 0             DM_Vy    126.8399   27595.9840  subfind_density()/subfind_density.c/96
 116 2973 0         FOF_PList    380.5665   18861.3858  fof_fof()/fof.c/249
 116 2974 0         FOF_GList    634.2775   19495.6633  fof_fof()/fof.c/337
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 10
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 108
 112 2981 0           Ngblist     63.4169   26866.3421  subfind_density()/subfind_density.c/78
 112 2982 0         Dist2list    126.8337   26993.1758  subfind_density()/subfind_density.c/79
 121 2991 0    	  DensDataGet    121.7456   27846.1604  subfind_density()/subfind_density.c/161
  28 2990 0             DM_Vz    126.8305   27724.5401  subfind_density()/subfind_density.c/97
 110 2991 0    	  DensDataGet    135.5615   27860.8051  subfind_density()/subfind_density.c/161
 110 2992 0  	  DensDataResult    101.6711   27962.4763  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=110: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=37.5237 MB).
  14 2991 0    	  DensDataGet    119.1501   27842.1669  subfind_density()/subfind_density.c/161
  14 2992 0  	  DensDataResult     89.3626   27931.5295  subfind_density()/subfind_density.c/201
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 106
  16 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  16 2973 0         FOF_PList    379.3390   18860.1582  fof_fof()/fof.c/249
  16 2974 0         FOF_GList    632.2316   19492.3898  fof_fof()/fof.c/337
 114 2990 0             DM_Vz    126.8399   27722.8239  subfind_density()/subfind_density.c/97
 116 2975 0             Group     11.3586   19507.0219  fof_fof()/fof.c/380
 116 2976 0   DomainNodeIndex      0.3473   19507.3692  force_treeallocate()/forcetree.c/4800
 116 2977 0        Nodes_base   3737.5399   23244.9090  force_treeallocate()/forcetree.c/4803
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 112 2983 0    DataIndexTable     21.8182   27014.9940  subfind_density()/subfind_density.c/87
 112 2984 0      DataNodeList     58.1818   27073.1758  subfind_density()/subfind_density.c/89
 112 2985 0              Left    126.8337   27200.0095  subfind_density()/subfind_density.c/91
 121 2992 0  	  DensDataResult     91.3092   27937.4697  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  28 2991 0    	  DensDataGet    136.2037   27860.7437  subfind_density()/subfind_density.c/161
task 110: endrun called with an error level of 812


----------------------------------------------------------------------------------------
  16 2975 0             Group     11.7020   19504.0918  fof_fof()/fof.c/380
  16 2976 0   DomainNodeIndex      0.3473   19504.4391  force_treeallocate()/forcetree.c/4800
  16 2977 0        Nodes_base   3737.5399   23241.9790  force_treeallocate()/forcetree.c/4803
 114 2991 0    	  DensDataGet    137.9637   27860.7877  subfind_density()/subfind_density.c/161
 116 2978 0     Extnodes_base   3114.6165   26359.5256  force_treeallocate()/forcetree.c/4811
 116 2979 0          Nextnode    222.3025   26581.8281  force_treeallocate()/forcetree.c/4820
 116 2980 0            Father    221.9056   26803.7337  force_treeallocate()/forcetree.c/4827
  11 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  11 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  11 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  11 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  11 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  11 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  11 2973 0         FOF_PList    380.5204   18861.3396  fof_fof()/fof.c/249
  11 2974 0         FOF_GList    634.2007   19495.5403  fof_fof()/fof.c/337
  11 2975 0             Group     12.7375   19508.2779  fof_fof()/fof.c/380
  11 2976 0   DomainNodeIndex      0.3473   19508.6252  force_treeallocate()/forcetree.c/4800
 112 2986 0             Right    126.8337   27326.8432  subfind_density()/subfind_density.c/92

Task=121: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.5303 MB).
task 121: endrun called with an error level of 812


  28 2992 0  	  DensDataResult    102.1528   27962.8965  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 110

Task=14: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.4705 MB).
  16 2978 0     Extnodes_base   3114.6165   26356.5955  force_treeallocate()/forcetree.c/4811
  16 2979 0          Nextnode    222.3025   26578.8980  force_treeallocate()/forcetree.c/4820
 114 2992 0  	  DensDataResult    103.4728   27964.2605  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 116 2981 0           Ngblist     63.4278   26867.1614  subfind_density()/subfind_density.c/78
 116 2982 0         Dist2list    126.8555   26994.0169  subfind_density()/subfind_density.c/79
  11 2977 0        Nodes_base   3737.5399   23246.1650  force_treeallocate()/forcetree.c/4803
  11 2978 0     Extnodes_base   3114.6165   26360.7816  force_treeallocate()/forcetree.c/4811
  11 2979 0          Nextnode    222.3025   26583.0841  force_treeallocate()/forcetree.c/4820
  11 2980 0            Father    221.9056   26804.9897  force_treeallocate()/forcetree.c/4827
 112 2987 0              Todo     15.8542   27342.6974  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 121

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
task 14: endrun called with an error level of 812


  16 2980 0            Father    221.9056   26800.8036  force_treeallocate()/forcetree.c/4827
  16 2981 0           Ngblist     63.2232   26864.0268  subfind_density()/subfind_density.c/78
  16 2982 0         Dist2list    126.4463   26990.4731  subfind_density()/subfind_density.c/79

Task=114: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=35.7395 MB).
task 114: endrun called with an error level of 812


 116 2983 0    DataIndexTable     21.8182   27015.8351  subfind_density()/subfind_density.c/87
  11 2981 0           Ngblist     63.4201   26868.4097  subfind_density()/subfind_density.c/78
  11 2982 0         Dist2list    126.8401   26995.2499  subfind_density()/subfind_density.c/79
  11 2983 0    DataIndexTable     21.8182   27017.0680  subfind_density()/subfind_density.c/87
 112 2988 0             DM_Vx    126.8337   27469.5311  subfind_density()/subfind_density.c/95

Task=28: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=37.1035 MB).
 111 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 111 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 111 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 111 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 111 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 111 2973 0         FOF_PList    380.5103   18861.3295  fof_fof()/fof.c/249
 111 2974 0         FOF_GList    634.1838   19495.5133  fof_fof()/fof.c/337
 111 2975 0             Group     11.8950   19507.4084  fof_fof()/fof.c/380
 111 2976 0   DomainNodeIndex      0.3473   19507.7557  force_treeallocate()/forcetree.c/4800
 111 2977 0        Nodes_base   3737.5399   23245.2955  force_treeallocate()/forcetree.c/4803
 111 2978 0     Extnodes_base   3114.6165   26359.9121  force_treeallocate()/forcetree.c/4811
 111 2979 0          Nextnode    222.3025   26582.2146  force_treeallocate()/forcetree.c/4820
 111 2980 0            Father    221.9056   26804.1202  force_treeallocate()/forcetree.c/4827
 111 2981 0           Ngblist     63.4184   26867.5385  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 14
  16 2983 0    DataIndexTable     21.8182   27012.2913  subfind_density()/subfind_density.c/87
  16 2984 0      DataNodeList     58.1818   27070.4731  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 114
 116 2984 0      DataNodeList     58.1818   27074.0169  subfind_density()/subfind_density.c/89
  11 2984 0      DataNodeList     58.1818   27075.2498  subfind_density()/subfind_density.c/89
  11 2985 0              Left    126.8401   27202.0900  subfind_density()/subfind_density.c/91
  11 2986 0             Right    126.8401   27328.9301  subfind_density()/subfind_density.c/92
 112 2989 0             DM_Vy    126.8337   27596.3648  subfind_density()/subfind_density.c/96
task 28: endrun called with an error level of 812


 111 2982 0         Dist2list    126.8368   26994.3753  subfind_density()/subfind_density.c/79
 111 2983 0    DataIndexTable     21.8182   27016.1935  subfind_density()/subfind_density.c/87
 111 2984 0      DataNodeList     58.1818   27074.3753  subfind_density()/subfind_density.c/89
 111 2985 0              Left    126.8368   27201.2120  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2985 0              Left    126.4463   27196.9194  subfind_density()/subfind_density.c/91
 116 2985 0              Left    126.8555   27200.8724  subfind_density()/subfind_density.c/91
  11 2987 0              Todo     15.8550   27344.7851  subfind_density()/subfind_density.c/93
  11 2988 0             DM_Vx    126.8401   27471.6253  subfind_density()/subfind_density.c/95
  11 2989 0             DM_Vy    126.8401   27598.4654  subfind_density()/subfind_density.c/96
  11 2990 0             DM_Vz    126.8401   27725.3055  subfind_density()/subfind_density.c/97
 112 2990 0             DM_Vz    126.8337   27723.1985  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 28
 111 2986 0             Right    126.8368   27328.0488  subfind_density()/subfind_density.c/92
 111 2987 0              Todo     15.8546   27343.9034  subfind_density()/subfind_density.c/93
 111 2988 0             DM_Vx    126.8368   27470.7402  subfind_density()/subfind_density.c/95
 111 2989 0             DM_Vy    126.8368   27597.5769  subfind_density()/subfind_density.c/96
  15 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  15 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  15 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  15 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  15 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  15 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  15 2973 0         FOF_PList    380.5890   18861.4083  fof_fof()/fof.c/249
  15 2974 0         FOF_GList    634.3151   19495.7234  fof_fof()/fof.c/337
  15 2975 0             Group     12.3817   19508.1050  fof_fof()/fof.c/380
  15 2976 0   DomainNodeIndex      0.3473   19508.4523  force_treeallocate()/forcetree.c/4800
  15 2977 0        Nodes_base   3737.5399   23245.9922  force_treeallocate()/forcetree.c/4803
  15 2978 0     Extnodes_base   3114.6165   26360.6087  force_treeallocate()/forcetree.c/4811
  16 2986 0             Right    126.4463   27323.3657  subfind_density()/subfind_density.c/92
 116 2986 0             Right    126.8555   27327.7279  subfind_density()/subfind_density.c/92
  11 2991 0    	  DensDataGet    112.0179   27837.3234  subfind_density()/subfind_density.c/161
 112 2991 0    	  DensDataGet    147.3862   27870.5847  subfind_density()/subfind_density.c/161
 112 2992 0  	  DensDataResult    110.5396   27981.1243  subfind_density()/subfind_density.c/201
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 111 2990 0             DM_Vz    126.8368   27724.4137  subfind_density()/subfind_density.c/97
 111 2991 0    	  DensDataGet    115.7001   27840.1137  subfind_density()/subfind_density.c/161
  15 2979 0          Nextnode    222.3025   26582.9112  force_treeallocate()/forcetree.c/4820
  15 2980 0            Father    221.9056   26804.8168  force_treeallocate()/forcetree.c/4827
  15 2981 0           Ngblist     63.4315   26868.2483  subfind_density()/subfind_density.c/78
  15 2982 0         Dist2list    126.8630   26995.1113  subfind_density()/subfind_density.c/79
  15 2983 0    DataIndexTable     21.8182   27016.9295  subfind_density()/subfind_density.c/87
  16 2987 0              Todo     15.8058   27339.1715  subfind_density()/subfind_density.c/93
 116 2987 0              Todo     15.8569   27343.5849  subfind_density()/subfind_density.c/93
  11 2992 0  	  DensDataResult     84.0134   27921.3368  subfind_density()/subfind_density.c/201
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  98 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  98 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  98 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  98 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  98 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2992 0  	  DensDataResult     86.7751   27926.8888  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=111: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.1112 MB).
task 111: endrun called with an error level of 812


  15 2984 0      DataNodeList     58.1818   27075.1113  subfind_density()/subfind_density.c/89
  15 2985 0              Left    126.8630   27201.9743  subfind_density()/subfind_density.c/91
  15 2986 0             Right    126.8630   27328.8373  subfind_density()/subfind_density.c/92
  15 2987 0              Todo     15.8579   27344.6952  subfind_density()/subfind_density.c/93
  16 2988 0             DM_Vx    126.4463   27465.6178  subfind_density()/subfind_density.c/95
 116 2988 0             DM_Vx    126.8555   27470.4404  subfind_density()/subfind_density.c/95


Task=112: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=18.8757 MB).
  98 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  98 2973 0         FOF_PList    380.5692   18861.3884  fof_fof()/fof.c/249
  98 2974 0         FOF_GList    634.2819   19495.6703  fof_fof()/fof.c/337
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 111
  15 2988 0             DM_Vx    126.8630   27471.5582  subfind_density()/subfind_density.c/95
  15 2989 0             DM_Vy    126.8630   27598.4212  subfind_density()/subfind_density.c/96
  15 2990 0             DM_Vz    126.8630   27725.2843  subfind_density()/subfind_density.c/97
  16 2989 0             DM_Vy    126.4463   27592.0641  subfind_density()/subfind_density.c/96
 116 2989 0             DM_Vy    126.8555   27597.2959  subfind_density()/subfind_density.c/96

Task=11: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.6632 MB).
task 11: endrun called with an error level of 812


task 112: endrun called with an error level of 812


  98 2975 0             Group     11.6771   19507.3474  fof_fof()/fof.c/380
  98 2976 0   DomainNodeIndex      0.3473   19507.6947  force_treeallocate()/forcetree.c/4800
  98 2977 0        Nodes_base   3737.5399   23245.2346  force_treeallocate()/forcetree.c/4803
  15 2991 0    	  DensDataGet    123.2277   27848.5120  subfind_density()/subfind_density.c/161
  15 2992 0  	  DensDataResult     92.4208   27940.9328  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  16 2990 0             DM_Vz    126.4463   27718.5105  subfind_density()/subfind_density.c/97
 116 2990 0             DM_Vz    126.8555   27724.1514  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 11
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 112
  98 2978 0     Extnodes_base   3114.6165   26359.8512  force_treeallocate()/forcetree.c/4811
  98 2979 0          Nextnode    222.3025   26582.1537  force_treeallocate()/forcetree.c/4820

Task=15: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.0672 MB).
task 15: endrun called with an error level of 812


  16 2991 0    	  DensDataGet    124.7866   27843.2970  subfind_density()/subfind_density.c/161
 116 2991 0    	  DensDataGet    119.8837   27844.0351  subfind_density()/subfind_density.c/161
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 112:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
  98 2980 0            Father    221.9056   26804.0592  force_treeallocate()/forcetree.c/4827
  98 2981 0           Ngblist     63.4282   26867.4874  subfind_density()/subfind_density.c/78
  98 2982 0         Dist2list    126.8564   26994.3438  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 15
  16 2992 0  	  DensDataResult     93.5899   27936.8869  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 116 2992 0  	  DensDataResult     89.9128   27933.9479  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 112: endrun called with an error level of 124


  98 2983 0    DataIndexTable     21.8182   27016.1620  subfind_density()/subfind_density.c/87
  98 2984 0      DataNodeList     58.1818   27074.3438  subfind_density()/subfind_density.c/89
  98 2985 0              Left    126.8564   27201.2002  subfind_density()/subfind_density.c/91

Task=16: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=63.1131 MB).
task 16: endrun called with an error level of 812



Task=116: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.0521 MB).
task 116: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 112
  98 2986 0             Right    126.8564   27328.0566  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 16
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 116
  98 2987 0              Todo     15.8570   27343.9136  subfind_density()/subfind_density.c/93
task 16: endrun called with an error level of 124


  98 2988 0             DM_Vx    126.8564   27470.7700  subfind_density()/subfind_density.c/95
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 16:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 16
  98 2989 0             DM_Vy    126.8564   27597.6264  subfind_density()/subfind_density.c/96
  98 2990 0             DM_Vz    126.8564   27724.4828  subfind_density()/subfind_density.c/97
  98 2991 0    	  DensDataGet    115.3449   27839.8277  subfind_density()/subfind_density.c/161
  98 2992 0  	  DensDataResult     86.5087   27926.3363  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=98: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.6637 MB).
task 98: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 98
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 109 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 109 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 109 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 109 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 109 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 109 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 109 2973 0         FOF_PList    380.4250   18861.2442  fof_fof()/fof.c/249
 109 2974 0         FOF_GList    634.0417   19495.2859  fof_fof()/fof.c/337
 109 2975 0             Group     11.4696   19506.7555  fof_fof()/fof.c/380
 109 2976 0   DomainNodeIndex      0.3473   19507.1028  force_treeallocate()/forcetree.c/4800
 109 2977 0        Nodes_base   3737.5399   23244.6427  force_treeallocate()/forcetree.c/4803
 109 2978 0     Extnodes_base   3114.6165   26359.2592  force_treeallocate()/forcetree.c/4811
 109 2979 0          Nextnode    222.3025   26581.5617  force_treeallocate()/forcetree.c/4820
 109 2980 0            Father    221.9056   26803.4673  force_treeallocate()/forcetree.c/4827
  18 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  18 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  18 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  18 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  18 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 109 2981 0           Ngblist     63.4042   26866.8715  subfind_density()/subfind_density.c/78
 109 2982 0         Dist2list    126.8083   26993.6798  subfind_density()/subfind_density.c/79
 109 2983 0    DataIndexTable     21.8182   27015.4980  subfind_density()/subfind_density.c/87
  18 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  18 2973 0         FOF_PList    380.4777   18861.2969  fof_fof()/fof.c/249
 109 2984 0      DataNodeList     58.1818   27073.6798  subfind_density()/subfind_density.c/89
 109 2985 0              Left    126.8083   27200.4881  subfind_density()/subfind_density.c/91
 109 2986 0             Right    126.8083   27327.2965  subfind_density()/subfind_density.c/92
 109 2987 0              Todo     15.8510   27343.1475  subfind_density()/subfind_density.c/93
  18 2974 0         FOF_GList    634.1295   19495.4264  fof_fof()/fof.c/337
  18 2975 0             Group     13.0374   19508.4638  fof_fof()/fof.c/380
  18 2976 0   DomainNodeIndex      0.3473   19508.8111  force_treeallocate()/forcetree.c/4800
 109 2988 0             DM_Vx    126.8083   27469.9558  subfind_density()/subfind_density.c/95
 109 2989 0             DM_Vy    126.8083   27596.7642  subfind_density()/subfind_density.c/96
 109 2990 0             DM_Vz    126.8083   27723.5725  subfind_density()/subfind_density.c/97
  18 2977 0        Nodes_base   3737.5399   23246.3510  force_treeallocate()/forcetree.c/4803
  18 2978 0     Extnodes_base   3114.6165   26360.9675  force_treeallocate()/forcetree.c/4811
 109 2991 0    	  DensDataGet    138.6063   27862.1788  subfind_density()/subfind_density.c/161
 109 2992 0  	  DensDataResult    103.9547   27966.1335  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  18 2979 0          Nextnode    222.3025   26583.2701  force_treeallocate()/forcetree.c/4820
  18 2980 0            Father    221.9056   26805.1756  force_treeallocate()/forcetree.c/4827
  18 2981 0           Ngblist     63.4129   26868.5886  subfind_density()/subfind_density.c/78

Task=109: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=33.8665 MB).
task 109: endrun called with an error level of 812


  18 2982 0         Dist2list    126.8259   26995.4145  subfind_density()/subfind_density.c/79
  18 2983 0    DataIndexTable     21.8182   27017.2327  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 109
  18 2984 0      DataNodeList     58.1818   27075.4145  subfind_density()/subfind_density.c/89
  18 2985 0              Left    126.8259   27202.2403  subfind_density()/subfind_density.c/91
  18 2986 0             Right    126.8259   27329.0662  subfind_density()/subfind_density.c/92
  18 2987 0              Todo     15.8532   27344.9195  subfind_density()/subfind_density.c/93
  18 2988 0             DM_Vx    126.8259   27471.7454  subfind_density()/subfind_density.c/95
  18 2989 0             DM_Vy    126.8259   27598.5713  subfind_density()/subfind_density.c/96
  18 2990 0             DM_Vz    126.8259   27725.3972  subfind_density()/subfind_density.c/97
  18 2991 0    	  DensDataGet    122.7584   27848.1556  subfind_density()/subfind_density.c/161
  18 2992 0  	  DensDataResult     92.0688   27940.2244  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=18: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.7756 MB).
task 18: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 18
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 107 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 107 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 107 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 107 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 107 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 107 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 107 2973 0         FOF_PList    380.4673   18861.2866  fof_fof()/fof.c/249
 107 2974 0         FOF_GList    634.1122   19495.3988  fof_fof()/fof.c/337
 107 2975 0             Group     11.9625   19507.3612  fof_fof()/fof.c/380
 107 2976 0   DomainNodeIndex      0.3473   19507.7085  force_treeallocate()/forcetree.c/4800
 107 2977 0        Nodes_base   3737.5399   23245.2484  force_treeallocate()/forcetree.c/4803
 107 2978 0     Extnodes_base   3114.6165   26359.8649  force_treeallocate()/forcetree.c/4811
 107 2979 0          Nextnode    222.3025   26582.1674  force_treeallocate()/forcetree.c/4820
 107 2980 0            Father    221.9056   26804.0730  force_treeallocate()/forcetree.c/4827
 107 2981 0           Ngblist     63.4112   26867.4843  subfind_density()/subfind_density.c/78
 107 2982 0         Dist2list    126.8224   26994.3067  subfind_density()/subfind_density.c/79
 107 2983 0    DataIndexTable     21.8182   27016.1249  subfind_density()/subfind_density.c/87
 107 2984 0      DataNodeList     58.1818   27074.3067  subfind_density()/subfind_density.c/89
 107 2985 0              Left    126.8224   27201.1291  subfind_density()/subfind_density.c/91
 107 2986 0             Right    126.8224   27327.9516  subfind_density()/subfind_density.c/92
 107 2987 0              Todo     15.8528   27343.8044  subfind_density()/subfind_density.c/93
 107 2988 0             DM_Vx    126.8224   27470.6268  subfind_density()/subfind_density.c/95
 107 2989 0             DM_Vy    126.8224   27597.4492  subfind_density()/subfind_density.c/96
 107 2990 0             DM_Vz    126.8224   27724.2717  subfind_density()/subfind_density.c/97
 107 2991 0    	  DensDataGet    108.3315   27832.6032  subfind_density()/subfind_density.c/161
 107 2992 0  	  DensDataResult     81.2486   27913.8518  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=107: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.1482 MB).
task 107: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 107
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 120 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 120 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 120 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 120 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 120 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2973 0         FOF_PList    380.4082   18861.2274  fof_fof()/fof.c/249
   7 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   7 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   7 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   7 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   7 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
   7 2973 0         FOF_PList    380.5962   18861.4155  fof_fof()/fof.c/249
   7 2974 0         FOF_GList    634.3270   19495.7425  fof_fof()/fof.c/337
   7 2975 0             Group     12.6058   19508.3483  fof_fof()/fof.c/380
   7 2976 0   DomainNodeIndex      0.3473   19508.6956  force_treeallocate()/forcetree.c/4800
   7 2977 0        Nodes_base   3737.5399   23246.2355  force_treeallocate()/forcetree.c/4803
   7 2978 0     Extnodes_base   3114.6165   26360.8520  force_treeallocate()/forcetree.c/4811
   7 2979 0          Nextnode    222.3025   26583.1545  force_treeallocate()/forcetree.c/4820
   7 2980 0            Father    221.9056   26805.0601  force_treeallocate()/forcetree.c/4827
   7 2981 0           Ngblist     63.4327   26868.4928  subfind_density()/subfind_density.c/78
   7 2982 0         Dist2list    126.8654   26995.3582  subfind_density()/subfind_density.c/79
   7 2983 0    DataIndexTable     21.8182   27017.1764  subfind_density()/subfind_density.c/87
 120 2974 0         FOF_GList    634.0136   19495.2410  fof_fof()/fof.c/337
 120 2975 0             Group     11.6833   19506.9243  fof_fof()/fof.c/380
 120 2976 0   DomainNodeIndex      0.3473   19507.2717  force_treeallocate()/forcetree.c/4800
   7 2984 0      DataNodeList     58.1818   27075.3582  subfind_density()/subfind_density.c/89
   7 2985 0              Left    126.8654   27202.2236  subfind_density()/subfind_density.c/91
   7 2986 0             Right    126.8654   27329.0890  subfind_density()/subfind_density.c/92
   7 2987 0              Todo     15.8582   27344.9472  subfind_density()/subfind_density.c/93
   7 2988 0             DM_Vx    126.8654   27471.8126  subfind_density()/subfind_density.c/95
   7 2989 0             DM_Vy    126.8654   27598.6780  subfind_density()/subfind_density.c/96
 120 2977 0        Nodes_base   3737.5399   23244.8115  force_treeallocate()/forcetree.c/4803
 120 2978 0     Extnodes_base   3114.6165   26359.4281  force_treeallocate()/forcetree.c/4811
   7 2990 0             DM_Vz    126.8654   27725.5434  subfind_density()/subfind_density.c/97
   7 2991 0    	  DensDataGet    107.4970   27833.0404  subfind_density()/subfind_density.c/161
   7 2992 0  	  DensDataResult     80.6228   27913.6632  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 120 2979 0          Nextnode    222.3025   26581.7306  force_treeallocate()/forcetree.c/4820

Task=7: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.3368 MB).
task 7: endrun called with an error level of 812


 120 2980 0            Father    221.9056   26803.6361  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 7
 120 2981 0           Ngblist     63.4014   26867.0375  subfind_density()/subfind_density.c/78
 120 2982 0         Dist2list    126.8027   26993.8402  subfind_density()/subfind_density.c/79
 120 2983 0    DataIndexTable     21.8182   27015.6584  subfind_density()/subfind_density.c/87
 120 2984 0      DataNodeList     58.1818   27073.8402  subfind_density()/subfind_density.c/89
 120 2985 0              Left    126.8027   27200.6429  subfind_density()/subfind_density.c/91
 120 2986 0             Right    126.8027   27327.4456  subfind_density()/subfind_density.c/92
 120 2987 0              Todo     15.8503   27343.2960  subfind_density()/subfind_density.c/93
 120 2988 0             DM_Vx    126.8027   27470.0987  subfind_density()/subfind_density.c/95
 120 2989 0             DM_Vy    126.8027   27596.9014  subfind_density()/subfind_density.c/96
 120 2990 0             DM_Vz    126.8027   27723.7041  subfind_density()/subfind_density.c/97
 120 2991 0    	  DensDataGet    109.3822   27833.0863  subfind_density()/subfind_density.c/161
 120 2992 0  	  DensDataResult     82.0367   27915.1230  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=120: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.877 MB).
task 120: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 120
task 120: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 120:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 120
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 105 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 105 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 105 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 105 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 105 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 105 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 105 2973 0         FOF_PList    380.4751   18861.2943  fof_fof()/fof.c/249
 105 2974 0         FOF_GList    634.1251   19495.4194  fof_fof()/fof.c/337
 105 2975 0             Group     11.6491   19507.0685  fof_fof()/fof.c/380
  22 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  22 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  22 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  22 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  22 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  22 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  22 2973 0         FOF_PList    380.5145   18861.3338  fof_fof()/fof.c/249
  22 2974 0         FOF_GList    634.1909   19495.5247  fof_fof()/fof.c/337
  22 2975 0             Group     10.6125   19506.1372  fof_fof()/fof.c/380
  22 2976 0   DomainNodeIndex      0.3473   19506.4845  force_treeallocate()/forcetree.c/4800
  22 2977 0        Nodes_base   3737.5399   23244.0244  force_treeallocate()/forcetree.c/4803
 105 2976 0   DomainNodeIndex      0.3473   19507.4158  force_treeallocate()/forcetree.c/4800
 105 2977 0        Nodes_base   3737.5399   23244.9557  force_treeallocate()/forcetree.c/4803
 105 2978 0     Extnodes_base   3114.6165   26359.5722  force_treeallocate()/forcetree.c/4811
 105 2979 0          Nextnode    222.3025   26581.8747  force_treeallocate()/forcetree.c/4820
 105 2980 0            Father    221.9056   26803.7803  force_treeallocate()/forcetree.c/4827
  22 2978 0     Extnodes_base   3114.6165   26358.6409  force_treeallocate()/forcetree.c/4811
  22 2979 0          Nextnode    222.3025   26580.9435  force_treeallocate()/forcetree.c/4820
  22 2980 0            Father    221.9056   26802.8490  force_treeallocate()/forcetree.c/4827
 105 2981 0           Ngblist     63.4125   26867.1928  subfind_density()/subfind_density.c/78
 105 2982 0         Dist2list    126.8250   26994.0178  subfind_density()/subfind_density.c/79
 105 2983 0    DataIndexTable     21.8182   27015.8360  subfind_density()/subfind_density.c/87
  22 2981 0           Ngblist     63.4191   26866.2681  subfind_density()/subfind_density.c/78
  22 2982 0         Dist2list    126.8382   26993.1063  subfind_density()/subfind_density.c/79
 105 2984 0      DataNodeList     58.1818   27074.0178  subfind_density()/subfind_density.c/89
 105 2985 0              Left    126.8250   27200.8428  subfind_density()/subfind_density.c/91
 105 2986 0             Right    126.8250   27327.6679  subfind_density()/subfind_density.c/92
  22 2983 0    DataIndexTable     21.8182   27014.9245  subfind_density()/subfind_density.c/87
  22 2984 0      DataNodeList     58.1818   27073.1063  subfind_density()/subfind_density.c/89
  22 2985 0              Left    126.8382   27199.9445  subfind_density()/subfind_density.c/91
 105 2987 0              Todo     15.8531   27343.5210  subfind_density()/subfind_density.c/93
 105 2988 0             DM_Vx    126.8250   27470.3460  subfind_density()/subfind_density.c/95
 105 2989 0             DM_Vy    126.8250   27597.1710  subfind_density()/subfind_density.c/96
  22 2986 0             Right    126.8382   27326.7826  subfind_density()/subfind_density.c/92
  22 2987 0              Todo     15.8548   27342.6374  subfind_density()/subfind_density.c/93
  22 2988 0             DM_Vx    126.8382   27469.4756  subfind_density()/subfind_density.c/95
 105 2990 0             DM_Vz    126.8250   27723.9960  subfind_density()/subfind_density.c/97
  22 2989 0             DM_Vy    126.8382   27596.3138  subfind_density()/subfind_density.c/96
 105 2991 0    	  DensDataGet    111.3624   27835.3584  subfind_density()/subfind_density.c/161
  22 2990 0             DM_Vz    126.8382   27723.1520  subfind_density()/subfind_density.c/97
 105 2992 0  	  DensDataResult     83.5218   27918.8802  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  22 2991 0    	  DensDataGet    145.4076   27868.5595  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=105: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.1198 MB).
task 105: endrun called with an error level of 812


  22 2992 0  	  DensDataResult    109.0557   27977.6152  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 123 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 123 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 123 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 123 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 123 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 123 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
 123 2973 0         FOF_PList    380.5384   18861.3577  fof_fof()/fof.c/249
 123 2974 0         FOF_GList    634.2307   19495.5884  fof_fof()/fof.c/337
 123 2975 0             Group     11.9168   19507.5052  fof_fof()/fof.c/380
 123 2976 0   DomainNodeIndex      0.3473   19507.8525  force_treeallocate()/forcetree.c/4800
 123 2977 0        Nodes_base   3737.5399   23245.3924  force_treeallocate()/forcetree.c/4803
 123 2978 0     Extnodes_base   3114.6165   26360.0089  force_treeallocate()/forcetree.c/4811
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 105

Task=22: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=22.3848 MB).
task 22: endrun called with an error level of 812


 123 2979 0          Nextnode    222.3025   26582.3114  force_treeallocate()/forcetree.c/4820
 123 2980 0            Father    221.9056   26804.2170  force_treeallocate()/forcetree.c/4827
 123 2981 0           Ngblist     63.4231   26867.6401  subfind_density()/subfind_density.c/78
 123 2982 0         Dist2list    126.8461   26994.4862  subfind_density()/subfind_density.c/79
 123 2983 0    DataIndexTable     21.8182   27016.3044  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 22
 123 2984 0      DataNodeList     58.1818   27074.4862  subfind_density()/subfind_density.c/89
 123 2985 0              Left    126.8461   27201.3324  subfind_density()/subfind_density.c/91
 123 2986 0             Right    126.8461   27328.1785  subfind_density()/subfind_density.c/92
 123 2987 0              Todo     15.8558   27344.0343  subfind_density()/subfind_density.c/93
 123 2988 0             DM_Vx    126.8461   27470.8804  subfind_density()/subfind_density.c/95
 123 2989 0             DM_Vy    126.8461   27597.7266  subfind_density()/subfind_density.c/96
 123 2990 0             DM_Vz    126.8461   27724.5727  subfind_density()/subfind_density.c/97
 123 2991 0    	  DensDataGet    130.4329   27855.0056  subfind_density()/subfind_density.c/161
 123 2992 0  	  DensDataResult     97.8247   27952.8303  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=123: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=47.1697 MB).
task 123: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 123
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 104 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 104 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 104 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 104 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 104 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 104 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  23 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  23 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  23 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  23 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  23 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  23 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  23 2973 0         FOF_PList    380.5010   18861.3202  fof_fof()/fof.c/249
  23 2974 0         FOF_GList    634.1683   19495.4885  fof_fof()/fof.c/337
  23 2975 0             Group     11.9666   19507.4551  fof_fof()/fof.c/380
  23 2976 0   DomainNodeIndex      0.3473   19507.8024  force_treeallocate()/forcetree.c/4800
  23 2977 0        Nodes_base   3737.5399   23245.3423  force_treeallocate()/forcetree.c/4803
  23 2978 0     Extnodes_base   3114.6165   26359.9588  force_treeallocate()/forcetree.c/4811
  23 2979 0          Nextnode    222.3025   26582.2613  force_treeallocate()/forcetree.c/4820
  23 2980 0            Father    221.9056   26804.1669  force_treeallocate()/forcetree.c/4827
 104 2973 0         FOF_PList    380.5808   18861.4000  fof_fof()/fof.c/249
 104 2974 0         FOF_GList    634.3013   19495.7013  fof_fof()/fof.c/337
 104 2975 0             Group     11.4042   19507.1055  fof_fof()/fof.c/380
  23 2981 0           Ngblist     63.4168   26867.5837  subfind_density()/subfind_density.c/78
  23 2982 0         Dist2list    126.8337   26994.4174  subfind_density()/subfind_density.c/79
  23 2983 0    DataIndexTable     21.8182   27016.2356  subfind_density()/subfind_density.c/87
  23 2984 0      DataNodeList     58.1818   27074.4174  subfind_density()/subfind_density.c/89
 104 2976 0   DomainNodeIndex      0.3473   19507.4528  force_treeallocate()/forcetree.c/4800
 104 2977 0        Nodes_base   3737.5399   23244.9927  force_treeallocate()/forcetree.c/4803
  23 2985 0              Left    126.8337   27201.2510  subfind_density()/subfind_density.c/91
  23 2986 0             Right    126.8337   27328.0847  subfind_density()/subfind_density.c/92
  23 2987 0              Todo     15.8542   27343.9389  subfind_density()/subfind_density.c/93
 104 2978 0     Extnodes_base   3114.6165   26359.6092  force_treeallocate()/forcetree.c/4811
 104 2979 0          Nextnode    222.3025   26581.9117  force_treeallocate()/forcetree.c/4820
  23 2988 0             DM_Vx    126.8337   27470.7725  subfind_density()/subfind_density.c/95
  23 2989 0             DM_Vy    126.8337   27597.6062  subfind_density()/subfind_density.c/96
  23 2990 0             DM_Vz    126.8337   27724.4399  subfind_density()/subfind_density.c/97
  23 2991 0    	  DensDataGet    128.7575   27853.1974  subfind_density()/subfind_density.c/161
 104 2980 0            Father    221.9056   26803.8173  force_treeallocate()/forcetree.c/4827
 104 2981 0           Ngblist     63.4301   26867.2474  subfind_density()/subfind_density.c/78
 104 2982 0         Dist2list    126.8603   26994.1077  subfind_density()/subfind_density.c/79
 104 2983 0    DataIndexTable     21.8182   27015.9259  subfind_density()/subfind_density.c/87
  23 2992 0  	  DensDataResult     96.5681   27949.7655  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=23: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.2345 MB).
task 23: endrun called with an error level of 812


 104 2984 0      DataNodeList     58.1818   27074.1077  subfind_density()/subfind_density.c/89
 104 2985 0              Left    126.8603   27200.9679  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 23
 104 2986 0             Right    126.8603   27327.8282  subfind_density()/subfind_density.c/92
 104 2987 0              Todo     15.8575   27343.6857  subfind_density()/subfind_density.c/93
 104 2988 0             DM_Vx    126.8603   27470.5459  subfind_density()/subfind_density.c/95
 104 2989 0             DM_Vy    126.8603   27597.4062  subfind_density()/subfind_density.c/96
 104 2990 0             DM_Vz    126.8603   27724.2664  subfind_density()/subfind_density.c/97
 104 2991 0    	  DensDataGet    114.6642   27838.9306  subfind_density()/subfind_density.c/161
 104 2992 0  	  DensDataResult     85.9981   27924.9288  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=104: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.0712 MB).
task 104: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 104
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 104:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 104: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 104
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  33 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  94 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  94 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  94 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  94 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  94 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  94 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  94 2973 0         FOF_PList    380.5295   18861.3487  fof_fof()/fof.c/249
  94 2974 0         FOF_GList    634.2158   19495.5645  fof_fof()/fof.c/337
  94 2975 0             Group     11.6865   19507.2510  fof_fof()/fof.c/380
  94 2976 0   DomainNodeIndex      0.3473   19507.5983  force_treeallocate()/forcetree.c/4800
  33 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  33 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  33 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  33 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  33 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  33 2973 0         FOF_PList    380.5218   18861.3410  fof_fof()/fof.c/249
  33 2974 0         FOF_GList    634.2030   19495.5440  fof_fof()/fof.c/337
  33 2975 0             Group     12.5995   19508.1435  fof_fof()/fof.c/380
  33 2976 0   DomainNodeIndex      0.3473   19508.4908  force_treeallocate()/forcetree.c/4800
  33 2977 0        Nodes_base   3737.5399   23246.0307  force_treeallocate()/forcetree.c/4803
  33 2978 0     Extnodes_base   3114.6165   26360.6472  force_treeallocate()/forcetree.c/4811
  94 2977 0        Nodes_base   3737.5399   23245.1382  force_treeallocate()/forcetree.c/4803
  94 2978 0     Extnodes_base   3114.6165   26359.7547  force_treeallocate()/forcetree.c/4811
  33 2979 0          Nextnode    222.3025   26582.9497  force_treeallocate()/forcetree.c/4820
  33 2980 0            Father    221.9056   26804.8553  force_treeallocate()/forcetree.c/4827
  33 2981 0           Ngblist     63.4203   26868.2756  subfind_density()/subfind_density.c/78
  33 2982 0         Dist2list    126.8406   26995.1162  subfind_density()/subfind_density.c/79
  33 2983 0    DataIndexTable     21.8182   27016.9344  subfind_density()/subfind_density.c/87
  94 2979 0          Nextnode    222.3025   26582.0572  force_treeallocate()/forcetree.c/4820
  94 2980 0            Father    221.9056   26803.9628  force_treeallocate()/forcetree.c/4827
  94 2981 0           Ngblist     63.4216   26867.3844  subfind_density()/subfind_density.c/78
  33 2984 0      DataNodeList     58.1818   27075.1162  subfind_density()/subfind_density.c/89
  33 2985 0              Left    126.8406   27201.9568  subfind_density()/subfind_density.c/91
  94 2982 0         Dist2list    126.8432   26994.2276  subfind_density()/subfind_density.c/79
  94 2983 0    DataIndexTable     21.8182   27016.0457  subfind_density()/subfind_density.c/87
  94 2984 0      DataNodeList     58.1818   27074.2275  subfind_density()/subfind_density.c/89
  33 2986 0             Right    126.8406   27328.7974  subfind_density()/subfind_density.c/92
  33 2987 0              Todo     15.8551   27344.6524  subfind_density()/subfind_density.c/93
  33 2988 0             DM_Vx    126.8406   27471.4930  subfind_density()/subfind_density.c/95
  94 2985 0              Left    126.8432   27201.0707  subfind_density()/subfind_density.c/91
  94 2986 0             Right    126.8432   27327.9138  subfind_density()/subfind_density.c/92
  94 2987 0              Todo     15.8554   27343.7692  subfind_density()/subfind_density.c/93
  33 2989 0             DM_Vy    126.8406   27598.3336  subfind_density()/subfind_density.c/96
  33 2990 0             DM_Vz    126.8406   27725.1742  subfind_density()/subfind_density.c/97
  94 2988 0             DM_Vx    126.8432   27470.6124  subfind_density()/subfind_density.c/95
  33 2991 0    	  DensDataGet    113.2347   27838.4090  subfind_density()/subfind_density.c/161
  33 2992 0  	  DensDataResult     84.9261   27923.3350  subfind_density()/subfind_density.c/201
  94 2989 0             DM_Vy    126.8432   27597.4556  subfind_density()/subfind_density.c/96
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  94 2990 0             DM_Vz    126.8432   27724.2987  subfind_density()/subfind_density.c/97

Task=33: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.665 MB).
  66 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  66 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  66 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  66 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  66 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  66 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  94 2991 0    	  DensDataGet    124.9678   27849.2666  subfind_density()/subfind_density.c/161
task 33: endrun called with an error level of 812


  66 2973 0         FOF_PList    380.1861   18861.0054  fof_fof()/fof.c/249
  66 2974 0         FOF_GList    633.6435   19494.6489  fof_fof()/fof.c/337
  66 2975 0             Group     11.5111   19506.1600  fof_fof()/fof.c/380
  94 2992 0  	  DensDataResult     93.7259   27942.9924  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 33
  66 2976 0   DomainNodeIndex      0.3473   19506.5073  force_treeallocate()/forcetree.c/4800
  66 2977 0        Nodes_base   3737.5399   23244.0472  force_treeallocate()/forcetree.c/4803

Task=94: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.0076 MB).
task 94: endrun called with an error level of 812


  66 2978 0     Extnodes_base   3114.6165   26358.6637  force_treeallocate()/forcetree.c/4811
  66 2979 0          Nextnode    222.3025   26580.9662  force_treeallocate()/forcetree.c/4820
  66 2980 0            Father    221.9056   26802.8718  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 94
  66 2981 0           Ngblist     63.3644   26866.2362  subfind_density()/subfind_density.c/78
  66 2982 0         Dist2list    126.7287   26992.9649  subfind_density()/subfind_density.c/79
  66 2983 0    DataIndexTable     21.8182   27014.7830  subfind_density()/subfind_density.c/87
  66 2984 0      DataNodeList     58.1818   27072.9648  subfind_density()/subfind_density.c/89
  66 2985 0              Left    126.7287   27199.6935  subfind_density()/subfind_density.c/91
  66 2986 0             Right    126.7287   27326.4222  subfind_density()/subfind_density.c/92
  66 2987 0              Todo     15.8411   27342.2633  subfind_density()/subfind_density.c/93
  66 2988 0             DM_Vx    126.7287   27468.9921  subfind_density()/subfind_density.c/95
  66 2989 0             DM_Vy    126.7287   27595.7208  subfind_density()/subfind_density.c/96
  66 2990 0             DM_Vz    126.7287   27722.4495  subfind_density()/subfind_density.c/97
  66 2991 0    	  DensDataGet    123.4036   27845.8530  subfind_density()/subfind_density.c/161
  66 2992 0  	  DensDataResult     92.5527   27938.4057  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=66: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.5943 MB).
task 66: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 66
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  12 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  12 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  12 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  12 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  12 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  12 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  12 2973 0         FOF_PList    380.5917   18861.4110  fof_fof()/fof.c/249
  12 2974 0         FOF_GList    634.3196   19495.7306  fof_fof()/fof.c/337
  12 2975 0             Group     12.8672   19508.5978  fof_fof()/fof.c/380
  12 2976 0   DomainNodeIndex      0.3473   19508.9451  force_treeallocate()/forcetree.c/4800
  12 2977 0        Nodes_base   3737.5399   23246.4850  force_treeallocate()/forcetree.c/4803
  12 2978 0     Extnodes_base   3114.6165   26361.1015  force_treeallocate()/forcetree.c/4811
  12 2979 0          Nextnode    222.3025   26583.4040  force_treeallocate()/forcetree.c/4820
  12 2980 0            Father    221.9056   26805.3096  force_treeallocate()/forcetree.c/4827
  12 2981 0           Ngblist     63.4320   26868.7416  subfind_density()/subfind_density.c/78
  12 2982 0         Dist2list    126.8639   26995.6055  subfind_density()/subfind_density.c/79
  12 2983 0    DataIndexTable     21.8182   27017.4237  subfind_density()/subfind_density.c/87
  12 2984 0      DataNodeList     58.1818   27075.6055  subfind_density()/subfind_density.c/89
  12 2985 0              Left    126.8639   27202.4694  subfind_density()/subfind_density.c/91
  12 2986 0             Right    126.8639   27329.3333  subfind_density()/subfind_density.c/92
  12 2987 0              Todo     15.8580   27345.1913  subfind_density()/subfind_density.c/93
  12 2988 0             DM_Vx    126.8639   27472.0552  subfind_density()/subfind_density.c/95
  12 2989 0             DM_Vy    126.8639   27598.9191  subfind_density()/subfind_density.c/96
  12 2990 0             DM_Vz    126.8639   27725.7830  subfind_density()/subfind_density.c/97
  12 2991 0    	  DensDataGet    109.3139   27835.0969  subfind_density()/subfind_density.c/161
  12 2992 0  	  DensDataResult     81.9854   27917.0824  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=12: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.9176 MB).
task 12: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 12
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  62 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  62 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  62 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  62 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  62 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  62 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  62 2973 0         FOF_PList    380.4540   18861.2732  fof_fof()/fof.c/249
  62 2974 0         FOF_GList    634.0900   19495.3633  fof_fof()/fof.c/337
  62 2975 0             Group     10.8325   19506.1958  fof_fof()/fof.c/380
  62 2976 0   DomainNodeIndex      0.3473   19506.5431  force_treeallocate()/forcetree.c/4800
  62 2977 0        Nodes_base   3737.5399   23244.0829  force_treeallocate()/forcetree.c/4803
  95 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  95 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  95 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  95 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  95 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  95 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  95 2973 0         FOF_PList    380.4682   18861.2874  fof_fof()/fof.c/249
  95 2974 0         FOF_GList    634.1137   19495.4011  fof_fof()/fof.c/337
  95 2975 0             Group     13.0457   19508.4468  fof_fof()/fof.c/380
  95 2976 0   DomainNodeIndex      0.3473   19508.7941  force_treeallocate()/forcetree.c/4800
  95 2977 0        Nodes_base   3737.5399   23246.3340  force_treeallocate()/forcetree.c/4803
  32 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  32 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  32 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  32 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  32 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  62 2978 0     Extnodes_base   3114.6165   26358.6995  force_treeallocate()/forcetree.c/4811
  62 2979 0          Nextnode    222.3025   26581.0020  force_treeallocate()/forcetree.c/4820
  95 2978 0     Extnodes_base   3114.6165   26360.9505  force_treeallocate()/forcetree.c/4811
  95 2979 0          Nextnode    222.3025   26583.2530  force_treeallocate()/forcetree.c/4820
  95 2980 0            Father    221.9056   26805.1586  force_treeallocate()/forcetree.c/4827
  32 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  32 2973 0         FOF_PList    380.6035   18861.4228  fof_fof()/fof.c/249
  62 2980 0            Father    221.9056   26802.9076  force_treeallocate()/forcetree.c/4827
  62 2981 0           Ngblist     63.4090   26866.3166  subfind_density()/subfind_density.c/78
  62 2982 0         Dist2list    126.8180   26993.1346  subfind_density()/subfind_density.c/79
  62 2983 0    DataIndexTable     21.8182   27014.9528  subfind_density()/subfind_density.c/87
  95 2981 0           Ngblist     63.4114   26868.5700  subfind_density()/subfind_density.c/78
  95 2982 0         Dist2list    126.8227   26995.3927  subfind_density()/subfind_density.c/79
  95 2983 0    DataIndexTable     21.8182   27017.2109  subfind_density()/subfind_density.c/87
  95 2984 0      DataNodeList     58.1818   27075.3927  subfind_density()/subfind_density.c/89
  32 2974 0         FOF_GList    634.3392   19495.7619  fof_fof()/fof.c/337
  32 2975 0             Group     12.0527   19507.8147  fof_fof()/fof.c/380
  32 2976 0   DomainNodeIndex      0.3473   19508.1620  force_treeallocate()/forcetree.c/4800
  62 2984 0      DataNodeList     58.1818   27073.1345  subfind_density()/subfind_density.c/89
  62 2985 0              Left    126.8180   27199.9525  subfind_density()/subfind_density.c/91
  62 2986 0             Right    126.8180   27326.7705  subfind_density()/subfind_density.c/92
  95 2985 0              Left    126.8227   27202.2154  subfind_density()/subfind_density.c/91
  95 2986 0             Right    126.8227   27329.0381  subfind_density()/subfind_density.c/92
  95 2987 0              Todo     15.8528   27344.8910  subfind_density()/subfind_density.c/93
  95 2988 0             DM_Vx    126.8227   27471.7137  subfind_density()/subfind_density.c/95
  32 2977 0        Nodes_base   3737.5399   23245.7018  force_treeallocate()/forcetree.c/4803
  32 2978 0     Extnodes_base   3114.6165   26360.3184  force_treeallocate()/forcetree.c/4811
  62 2987 0              Todo     15.8523   27342.6228  subfind_density()/subfind_density.c/93
  62 2988 0             DM_Vx    126.8180   27469.4408  subfind_density()/subfind_density.c/95
  62 2989 0             DM_Vy    126.8180   27596.2588  subfind_density()/subfind_density.c/96
  95 2989 0             DM_Vy    126.8227   27598.5364  subfind_density()/subfind_density.c/96
  95 2990 0             DM_Vz    126.8227   27725.3592  subfind_density()/subfind_density.c/97
  95 2991 0    	  DensDataGet    123.7829   27849.1421  subfind_density()/subfind_density.c/161
  32 2979 0          Nextnode    222.3025   26582.6209  force_treeallocate()/forcetree.c/4820
  32 2980 0            Father    221.9056   26804.5265  force_treeallocate()/forcetree.c/4827
  62 2990 0             DM_Vz    126.8180   27723.0768  subfind_density()/subfind_density.c/97
  95 2992 0  	  DensDataResult     92.8372   27941.9792  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  32 2981 0           Ngblist     63.4339   26867.9604  subfind_density()/subfind_density.c/78
  32 2982 0         Dist2list    126.8678   26994.8282  subfind_density()/subfind_density.c/79
  62 2991 0    	  DensDataGet    137.8133   27860.8901  subfind_density()/subfind_density.c/161

Task=95: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.0208 MB).
  32 2983 0    DataIndexTable     21.8182   27016.6464  subfind_density()/subfind_density.c/87
  62 2992 0  	  DensDataResult    103.3600   27964.2501  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 95: endrun called with an error level of 812


  32 2984 0      DataNodeList     58.1818   27074.8282  subfind_density()/subfind_density.c/89

Task=62: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=35.7499 MB).
task 62: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 95
  32 2985 0              Left    126.8678   27201.6960  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 62
  32 2986 0             Right    126.8678   27328.5639  subfind_density()/subfind_density.c/92
  32 2987 0              Todo     15.8585   27344.4223  subfind_density()/subfind_density.c/93
  32 2988 0             DM_Vx    126.8678   27471.2902  subfind_density()/subfind_density.c/95
  32 2989 0             DM_Vy    126.8678   27598.1580  subfind_density()/subfind_density.c/96
  32 2990 0             DM_Vz    126.8678   27725.0259  subfind_density()/subfind_density.c/97
  32 2991 0    	  DensDataGet    121.2179   27846.2438  subfind_density()/subfind_density.c/161
  32 2992 0  	  DensDataResult     90.9134   27937.1572  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=32: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.8428 MB).
task 32: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 32
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 32:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 32: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 32
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  34 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  34 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  34 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  34 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  34 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  34 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  34 2973 0         FOF_PList    380.0852   18860.9044  fof_fof()/fof.c/249
  34 2974 0         FOF_GList    633.4753   19494.3797  fof_fof()/fof.c/337
  34 2975 0             Group     13.5303   19507.9099  fof_fof()/fof.c/380
  34 2976 0   DomainNodeIndex      0.3473   19508.2572  force_treeallocate()/forcetree.c/4800
  34 2977 0        Nodes_base   3737.5399   23245.7971  force_treeallocate()/forcetree.c/4803
  34 2978 0     Extnodes_base   3114.6165   26360.4137  force_treeallocate()/forcetree.c/4811
  34 2979 0          Nextnode    222.3025   26582.7162  force_treeallocate()/forcetree.c/4820
  34 2980 0            Father    221.9056   26804.6217  force_treeallocate()/forcetree.c/4827
  34 2981 0           Ngblist     63.3475   26867.9693  subfind_density()/subfind_density.c/78
  34 2982 0         Dist2list    126.6951   26994.6643  subfind_density()/subfind_density.c/79
  34 2983 0    DataIndexTable     21.8182   27016.4825  subfind_density()/subfind_density.c/87
  34 2984 0      DataNodeList     58.1818   27074.6643  subfind_density()/subfind_density.c/89
  34 2985 0              Left    126.6951   27201.3593  subfind_density()/subfind_density.c/91
  34 2986 0             Right    126.6951   27328.0544  subfind_density()/subfind_density.c/92
  34 2987 0              Todo     15.8369   27343.8913  subfind_density()/subfind_density.c/93
  34 2988 0             DM_Vx    126.6951   27470.5863  subfind_density()/subfind_density.c/95
  34 2989 0             DM_Vy    126.6951   27597.2814  subfind_density()/subfind_density.c/96
  34 2990 0             DM_Vz    126.6951   27723.9764  subfind_density()/subfind_density.c/97
  34 2991 0    	  DensDataGet    109.9996   27833.9761  subfind_density()/subfind_density.c/161
  34 2992 0  	  DensDataResult     82.4997   27916.4758  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=34: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.5242 MB).
task 34: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 34
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  92 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  92 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  92 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  92 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  92 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  92 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  92 2973 0         FOF_PList    380.4212   18861.2405  fof_fof()/fof.c/249
  92 2974 0         FOF_GList    634.0354   19495.2759  fof_fof()/fof.c/337
  92 2975 0             Group     10.2587   19505.5346  fof_fof()/fof.c/380
  92 2976 0   DomainNodeIndex      0.3473   19505.8819  force_treeallocate()/forcetree.c/4800
  35 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  35 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  35 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  35 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  35 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  35 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  35 2973 0         FOF_PList    380.4838   18861.3031  fof_fof()/fof.c/249
  35 2974 0         FOF_GList    634.1397   19495.4428  fof_fof()/fof.c/337
  35 2975 0             Group     11.6335   19507.0763  fof_fof()/fof.c/380
  35 2976 0   DomainNodeIndex      0.3473   19507.4236  force_treeallocate()/forcetree.c/4800
  35 2977 0        Nodes_base   3737.5399   23244.9635  force_treeallocate()/forcetree.c/4803
  35 2978 0     Extnodes_base   3114.6165   26359.5800  force_treeallocate()/forcetree.c/4811
  35 2979 0          Nextnode    222.3025   26581.8825  force_treeallocate()/forcetree.c/4820
  35 2980 0            Father    221.9056   26803.7881  force_treeallocate()/forcetree.c/4827
  92 2977 0        Nodes_base   3737.5399   23243.4218  force_treeallocate()/forcetree.c/4803
  92 2978 0     Extnodes_base   3114.6165   26358.0383  force_treeallocate()/forcetree.c/4811
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  35 2981 0           Ngblist     63.4140   26867.2021  subfind_density()/subfind_density.c/78
  35 2982 0         Dist2list    126.8279   26994.0300  subfind_density()/subfind_density.c/79
  35 2983 0    DataIndexTable     21.8182   27015.8482  subfind_density()/subfind_density.c/87
  35 2984 0      DataNodeList     58.1818   27074.0300  subfind_density()/subfind_density.c/89
  92 2979 0          Nextnode    222.3025   26580.3409  force_treeallocate()/forcetree.c/4820
  92 2980 0            Father    221.9056   26802.2464  force_treeallocate()/forcetree.c/4827
  92 2981 0           Ngblist     63.4035   26865.6500  subfind_density()/subfind_density.c/78
  64 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  64 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  64 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  64 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  64 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  64 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  64 2973 0         FOF_PList    379.7048   18860.5240  fof_fof()/fof.c/249
  64 2974 0         FOF_GList    632.8413   19493.3653  fof_fof()/fof.c/337
  64 2975 0             Group     10.8512   19504.2165  fof_fof()/fof.c/380
  64 2976 0   DomainNodeIndex      0.3473   19504.5638  force_treeallocate()/forcetree.c/4800
  35 2985 0              Left    126.8279   27200.8579  subfind_density()/subfind_density.c/91
  35 2986 0             Right    126.8279   27327.6859  subfind_density()/subfind_density.c/92
  35 2987 0              Todo     15.8535   27343.5394  subfind_density()/subfind_density.c/93
  35 2988 0             DM_Vx    126.8279   27470.3673  subfind_density()/subfind_density.c/95
  92 2982 0         Dist2list    126.8071   26992.4571  subfind_density()/subfind_density.c/79
  92 2983 0    DataIndexTable     21.8182   27014.2752  subfind_density()/subfind_density.c/87
  92 2984 0      DataNodeList     58.1818   27072.4570  subfind_density()/subfind_density.c/89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  64 2977 0        Nodes_base   3737.5399   23242.1037  force_treeallocate()/forcetree.c/4803
  64 2978 0     Extnodes_base   3114.6165   26356.7202  force_treeallocate()/forcetree.c/4811
  35 2989 0             DM_Vy    126.8279   27597.1953  subfind_density()/subfind_density.c/96
  35 2990 0             DM_Vz    126.8279   27724.0232  subfind_density()/subfind_density.c/97
  35 2991 0    	  DensDataGet    109.6569   27833.6801  subfind_density()/subfind_density.c/161
  92 2985 0              Left    126.8071   27199.2641  subfind_density()/subfind_density.c/91
  92 2986 0             Right    126.8071   27326.0712  subfind_density()/subfind_density.c/92
  57 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  57 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  57 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  57 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  57 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  57 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  57 2973 0         FOF_PList    380.4516   18861.2709  fof_fof()/fof.c/249
  57 2974 0         FOF_GList    634.0861   19495.3570  fof_fof()/fof.c/337
  57 2975 0             Group     11.7539   19507.1109  fof_fof()/fof.c/380
  57 2976 0   DomainNodeIndex      0.3473   19507.4582  force_treeallocate()/forcetree.c/4800
  57 2977 0        Nodes_base   3737.5399   23244.9980  force_treeallocate()/forcetree.c/4803
  57 2978 0     Extnodes_base   3114.6165   26359.6146  force_treeallocate()/forcetree.c/4811
  57 2979 0          Nextnode    222.3025   26581.9171  force_treeallocate()/forcetree.c/4820
  57 2980 0            Father    221.9056   26803.8227  force_treeallocate()/forcetree.c/4827
  57 2981 0           Ngblist     63.4086   26867.2313  subfind_density()/subfind_density.c/78
  64 2979 0          Nextnode    222.3025   26579.0227  force_treeallocate()/forcetree.c/4820
  64 2980 0            Father    221.9056   26800.9283  force_treeallocate()/forcetree.c/4827
  64 2981 0           Ngblist     63.2841   26864.2125  subfind_density()/subfind_density.c/78
  35 2992 0  	  DensDataResult     82.2427   27915.9228  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=35: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.0772 MB).
task 35: endrun called with an error level of 812


  92 2987 0              Todo     15.8509   27341.9221  subfind_density()/subfind_density.c/93
  92 2988 0             DM_Vx    126.8071   27468.7292  subfind_density()/subfind_density.c/95
  57 2982 0         Dist2list    126.8172   26994.0485  subfind_density()/subfind_density.c/79
  57 2983 0    DataIndexTable     21.8182   27015.8667  subfind_density()/subfind_density.c/87
  57 2984 0      DataNodeList     58.1818   27074.0485  subfind_density()/subfind_density.c/89
  57 2985 0              Left    126.8172   27200.8657  subfind_density()/subfind_density.c/91
  57 2986 0             Right    126.8172   27327.6829  subfind_density()/subfind_density.c/92
  64 2982 0         Dist2list    126.5683   26990.7807  subfind_density()/subfind_density.c/79
  64 2983 0    DataIndexTable     21.8182   27012.5989  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 35
  92 2989 0             DM_Vy    126.8071   27595.5363  subfind_density()/subfind_density.c/96
  57 2987 0              Todo     15.8522   27343.5350  subfind_density()/subfind_density.c/93
  57 2988 0             DM_Vx    126.8172   27470.3523  subfind_density()/subfind_density.c/95
  57 2989 0             DM_Vy    126.8172   27597.1695  subfind_density()/subfind_density.c/96
  64 2984 0      DataNodeList     58.1818   27070.7807  subfind_density()/subfind_density.c/89
  64 2985 0              Left    126.5683   27197.3489  subfind_density()/subfind_density.c/91
  64 2986 0             Right    126.5683   27323.9172  subfind_density()/subfind_density.c/92
  64 2987 0              Todo     15.8210   27339.7382  subfind_density()/subfind_density.c/93
  92 2990 0             DM_Vz    126.8071   27722.3433  subfind_density()/subfind_density.c/97
  57 2990 0             DM_Vz    126.8172   27723.9867  subfind_density()/subfind_density.c/97
  57 2991 0    	  DensDataGet    118.9247   27842.9114  subfind_density()/subfind_density.c/161
  57 2992 0  	  DensDataResult     89.1935   27932.1049  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  64 2988 0             DM_Vx    126.5683   27466.3065  subfind_density()/subfind_density.c/95
  92 2991 0    	  DensDataGet    121.2624   27843.6057  subfind_density()/subfind_density.c/161

Task=57: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.8951 MB).
task 57: endrun called with an error level of 812


  64 2989 0             DM_Vy    126.5683   27592.8748  subfind_density()/subfind_density.c/96
  92 2992 0  	  DensDataResult     90.9468   27934.5525  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 57
  64 2990 0             DM_Vz    126.5683   27719.4430  subfind_density()/subfind_density.c/97

  64 2991 0    	  DensDataGet    130.9417   27850.3847  subfind_density()/subfind_density.c/161

Task=92: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.4475 MB).
task 92: endrun called with an error level of 812


  64 2992 0  	  DensDataResult     98.2063   27948.5910  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 92

Task=64: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.409 MB).
task 64: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 64
task 64: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 64:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 64
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  69 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  69 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  69 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  69 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  69 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  69 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  69 2973 0         FOF_PList    380.5982   18861.4174  fof_fof()/fof.c/249
  69 2974 0         FOF_GList    634.3303   19495.7477  fof_fof()/fof.c/337
  69 2975 0             Group     10.2587   19506.0064  fof_fof()/fof.c/380
  69 2976 0   DomainNodeIndex      0.3473   19506.3537  force_treeallocate()/forcetree.c/4800
  69 2977 0        Nodes_base   3737.5399   23243.8935  force_treeallocate()/forcetree.c/4803
  69 2978 0     Extnodes_base   3114.6165   26358.5101  force_treeallocate()/forcetree.c/4811
  69 2979 0          Nextnode    222.3025   26580.8126  force_treeallocate()/forcetree.c/4820
  69 2980 0            Father    221.9056   26802.7182  force_treeallocate()/forcetree.c/4827
  69 2981 0           Ngblist     63.4330   26866.1512  subfind_density()/subfind_density.c/78
  69 2982 0         Dist2list    126.8661   26993.0173  subfind_density()/subfind_density.c/79
  69 2983 0    DataIndexTable     21.8182   27014.8354  subfind_density()/subfind_density.c/87
  69 2984 0      DataNodeList     58.1818   27073.0172  subfind_density()/subfind_density.c/89
  69 2985 0              Left    126.8661   27199.8833  subfind_density()/subfind_density.c/91
  69 2986 0             Right    126.8661   27326.7493  subfind_density()/subfind_density.c/92
  69 2987 0              Todo     15.8583   27342.6076  subfind_density()/subfind_density.c/93
  69 2988 0             DM_Vx    126.8661   27469.4736  subfind_density()/subfind_density.c/95
  69 2989 0             DM_Vy    126.8661   27596.3397  subfind_density()/subfind_density.c/96
  69 2990 0             DM_Vz    126.8661   27723.2057  subfind_density()/subfind_density.c/97
  69 2991 0    	  DensDataGet    132.4243   27855.6300  subfind_density()/subfind_density.c/161
  69 2992 0  	  DensDataResult     99.3182   27954.9482  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=69: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.0518 MB).
task 69: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 69
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  56 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  56 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  56 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  56 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  56 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  56 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  56 2973 0         FOF_PList    380.4914   18861.3106  fof_fof()/fof.c/249
  56 2974 0         FOF_GList    634.1523   19495.4629  fof_fof()/fof.c/337
  56 2975 0             Group     11.5350   19506.9979  fof_fof()/fof.c/380
  56 2976 0   DomainNodeIndex      0.3473   19507.3452  force_treeallocate()/forcetree.c/4800
  56 2977 0        Nodes_base   3737.5399   23244.8851  force_treeallocate()/forcetree.c/4803
  56 2978 0     Extnodes_base   3114.6165   26359.5016  force_treeallocate()/forcetree.c/4811
  56 2979 0          Nextnode    222.3025   26581.8041  force_treeallocate()/forcetree.c/4820
  56 2980 0            Father    221.9056   26803.7097  force_treeallocate()/forcetree.c/4827
  56 2981 0           Ngblist     63.4152   26867.1249  subfind_density()/subfind_density.c/78
  56 2982 0         Dist2list    126.8305   26993.9554  subfind_density()/subfind_density.c/79
  56 2983 0    DataIndexTable     21.8182   27015.7736  subfind_density()/subfind_density.c/87
  56 2984 0      DataNodeList     58.1818   27073.9554  subfind_density()/subfind_density.c/89
  56 2985 0              Left    126.8305   27200.7858  subfind_density()/subfind_density.c/91
  56 2986 0             Right    126.8305   27327.6163  subfind_density()/subfind_density.c/92
  56 2987 0              Todo     15.8538   27343.4701  subfind_density()/subfind_density.c/93
  56 2988 0             DM_Vx    126.8305   27470.3005  subfind_density()/subfind_density.c/95
  56 2989 0             DM_Vy    126.8305   27597.1310  subfind_density()/subfind_density.c/96
  56 2990 0             DM_Vz    126.8305   27723.9615  subfind_density()/subfind_density.c/97
  56 2991 0    	  DensDataGet    110.2250   27834.1865  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  56 2992 0  	  DensDataResult     82.6688   27916.8553  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  51 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  51 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  51 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  51 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  51 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  51 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  51 2973 0         FOF_PList    380.5311   18861.3503  fof_fof()/fof.c/249
  51 2974 0         FOF_GList    634.2184   19495.5688  fof_fof()/fof.c/337
  51 2975 0             Group     11.2144   19506.7831  fof_fof()/fof.c/380
  51 2976 0   DomainNodeIndex      0.3473   19507.1304  force_treeallocate()/forcetree.c/4800
  51 2977 0        Nodes_base   3737.5399   23244.6703  force_treeallocate()/forcetree.c/4803
  51 2978 0     Extnodes_base   3114.6165   26359.2868  force_treeallocate()/forcetree.c/4811
  51 2979 0          Nextnode    222.3025   26581.5893  force_treeallocate()/forcetree.c/4820
  51 2980 0            Father    221.9056   26803.4949  force_treeallocate()/forcetree.c/4827
  51 2981 0           Ngblist     63.4218   26866.9168  subfind_density()/subfind_density.c/78
  67 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  67 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  67 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  67 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  67 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  67 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  67 2973 0         FOF_PList    380.5271   18861.3463  fof_fof()/fof.c/249
  67 2974 0         FOF_GList    634.2118   19495.5581  fof_fof()/fof.c/337
  67 2975 0             Group     11.0151   19506.5733  fof_fof()/fof.c/380
  67 2976 0   DomainNodeIndex      0.3473   19506.9206  force_treeallocate()/forcetree.c/4800
  67 2977 0        Nodes_base   3737.5399   23244.4604  force_treeallocate()/forcetree.c/4803
  67 2978 0     Extnodes_base   3114.6165   26359.0770  force_treeallocate()/forcetree.c/4811
  67 2979 0          Nextnode    222.3025   26581.3795  force_treeallocate()/forcetree.c/4820
  67 2980 0            Father    221.9056   26803.2851  force_treeallocate()/forcetree.c/4827
  67 2981 0           Ngblist     63.4212   26866.7063  subfind_density()/subfind_density.c/78

Task=56: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.1447 MB).
task 56: endrun called with an error level of 812


  51 2982 0         Dist2list    126.8437   26993.7604  subfind_density()/subfind_density.c/79
  51 2983 0    DataIndexTable     21.8182   27015.5786  subfind_density()/subfind_density.c/87
  51 2984 0      DataNodeList     58.1818   27073.7604  subfind_density()/subfind_density.c/89
  51 2985 0              Left    126.8437   27200.6041  subfind_density()/subfind_density.c/91
  51 2986 0             Right    126.8437   27327.4478  subfind_density()/subfind_density.c/92
  67 2982 0         Dist2list    126.8424   26993.5486  subfind_density()/subfind_density.c/79
  67 2983 0    DataIndexTable     21.8182   27015.3668  subfind_density()/subfind_density.c/87
  67 2984 0      DataNodeList     58.1818   27073.5486  subfind_density()/subfind_density.c/89
  67 2985 0              Left    126.8424   27200.3909  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 56
  51 2987 0              Todo     15.8555   27343.3033  subfind_density()/subfind_density.c/93
  51 2988 0             DM_Vx    126.8437   27470.1469  subfind_density()/subfind_density.c/95
  51 2989 0             DM_Vy    126.8437   27596.9906  subfind_density()/subfind_density.c/96
  51 2990 0             DM_Vz    126.8437   27723.8343  subfind_density()/subfind_density.c/97
  67 2986 0             Right    126.8424   27327.2333  subfind_density()/subfind_density.c/92
  67 2987 0              Todo     15.8553   27343.0886  subfind_density()/subfind_density.c/93
  67 2988 0             DM_Vx    126.8424   27469.9310  subfind_density()/subfind_density.c/95
  67 2989 0             DM_Vy    126.8424   27596.7733  subfind_density()/subfind_density.c/96
task 56: endrun called with an error level of 124


  51 2991 0    	  DensDataGet    108.7448   27832.5791  subfind_density()/subfind_density.c/161
  51 2992 0  	  DensDataResult     81.5586   27914.1377  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  67 2990 0             DM_Vz    126.8424   27723.6157  subfind_density()/subfind_density.c/97
  67 2991 0    	  DensDataGet    152.6509   27876.2666  subfind_density()/subfind_density.c/161
  67 2992 0  	  DensDataResult    114.4882   27990.7547  subfind_density()/subfind_density.c/201
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 56:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 56
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=51: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.8623 MB).
task 51: endrun called with an error level of 812


----------------------------------------------------------------------------------------

Task=67: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=9.24527 MB).
task 67: endrun called with an error level of 812


  68 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  68 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  68 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  68 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  68 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  68 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  68 2973 0         FOF_PList    380.4309   18861.2501  fof_fof()/fof.c/249
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 51
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 67
  68 2974 0         FOF_GList    634.0515   19495.3016  fof_fof()/fof.c/337
  68 2975 0             Group     11.4177   19506.7193  fof_fof()/fof.c/380
  68 2976 0   DomainNodeIndex      0.3473   19507.0666  force_treeallocate()/forcetree.c/4800
  68 2977 0        Nodes_base   3737.5399   23244.6065  force_treeallocate()/forcetree.c/4803
  68 2978 0     Extnodes_base   3114.6165   26359.2230  force_treeallocate()/forcetree.c/4811
  68 2979 0          Nextnode    222.3025   26581.5255  force_treeallocate()/forcetree.c/4820
  68 2980 0            Father    221.9056   26803.4311  force_treeallocate()/forcetree.c/4827
  68 2981 0           Ngblist     63.4052   26866.8363  subfind_density()/subfind_density.c/78
  68 2982 0         Dist2list    126.8103   26993.6466  subfind_density()/subfind_density.c/79
  68 2983 0    DataIndexTable     21.8182   27015.4648  subfind_density()/subfind_density.c/87
  68 2984 0      DataNodeList     58.1818   27073.6465  subfind_density()/subfind_density.c/89
  68 2985 0              Left    126.8103   27200.4568  subfind_density()/subfind_density.c/91
  68 2986 0             Right    126.8103   27327.2671  subfind_density()/subfind_density.c/92
  68 2987 0              Todo     15.8513   27343.1184  subfind_density()/subfind_density.c/93
  68 2988 0             DM_Vx    126.8103   27469.9287  subfind_density()/subfind_density.c/95
  68 2989 0             DM_Vy    126.8103   27596.7390  subfind_density()/subfind_density.c/96
  68 2990 0             DM_Vz    126.8103   27723.5493  subfind_density()/subfind_density.c/97
  68 2991 0    	  DensDataGet    123.4420   27846.9913  subfind_density()/subfind_density.c/161
  68 2992 0  	  DensDataResult     92.5815   27939.5727  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=68: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.4273 MB).
task 68: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 68
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  50 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  50 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  50 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  50 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  50 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  50 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  50 2973 0         FOF_PList    380.5316   18861.3509  fof_fof()/fof.c/249
  50 2974 0         FOF_GList    634.2194   19495.5702  fof_fof()/fof.c/337
  50 2975 0             Group     12.1762   19507.7464  fof_fof()/fof.c/380
  50 2976 0   DomainNodeIndex      0.3473   19508.0937  force_treeallocate()/forcetree.c/4800
  50 2977 0        Nodes_base   3737.5399   23245.6336  force_treeallocate()/forcetree.c/4803
  50 2978 0     Extnodes_base   3114.6165   26360.2501  force_treeallocate()/forcetree.c/4811
  50 2979 0          Nextnode    222.3025   26582.5526  force_treeallocate()/forcetree.c/4820
  50 2980 0            Father    221.9056   26804.4582  force_treeallocate()/forcetree.c/4827
  50 2981 0           Ngblist     63.4219   26867.8802  subfind_density()/subfind_density.c/78
  50 2982 0         Dist2list    126.8439   26994.7240  subfind_density()/subfind_density.c/79
  50 2983 0    DataIndexTable     21.8182   27016.5422  subfind_density()/subfind_density.c/87
  50 2984 0      DataNodeList     58.1818   27074.7240  subfind_density()/subfind_density.c/89
  50 2985 0              Left    126.8439   27201.5679  subfind_density()/subfind_density.c/91
  50 2986 0             Right    126.8439   27328.4118  subfind_density()/subfind_density.c/92
  50 2987 0              Todo     15.8555   27344.2672  subfind_density()/subfind_density.c/93
  50 2988 0             DM_Vx    126.8439   27471.1111  subfind_density()/subfind_density.c/95
  50 2989 0             DM_Vy    126.8439   27597.9550  subfind_density()/subfind_density.c/96
  50 2990 0             DM_Vz    126.8439   27724.7989  subfind_density()/subfind_density.c/97
  50 2991 0    	  DensDataGet    115.5800   27840.3788  subfind_density()/subfind_density.c/161
  50 2992 0  	  DensDataResult     86.6850   27927.0638  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=50: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.9362 MB).
task 50: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 50
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  87 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  87 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  87 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  87 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  87 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  87 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  87 2973 0         FOF_PList    380.1574   18860.9767  fof_fof()/fof.c/249
  87 2974 0         FOF_GList    633.5957   19494.5724  fof_fof()/fof.c/337
  87 2975 0             Group     12.7490   19507.3213  fof_fof()/fof.c/380
  87 2976 0   DomainNodeIndex      0.3473   19507.6686  force_treeallocate()/forcetree.c/4800
  87 2977 0        Nodes_base   3737.5399   23245.2085  force_treeallocate()/forcetree.c/4803
  87 2978 0     Extnodes_base   3114.6165   26359.8250  force_treeallocate()/forcetree.c/4811
  87 2979 0          Nextnode    222.3025   26582.1275  force_treeallocate()/forcetree.c/4820
  87 2980 0            Father    221.9056   26804.0331  force_treeallocate()/forcetree.c/4827
  87 2981 0           Ngblist     63.3596   26867.3927  subfind_density()/subfind_density.c/78
  40 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  40 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  40 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  40 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  40 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  40 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  40 2973 0         FOF_PList    380.2702   18861.0895  fof_fof()/fof.c/249
  40 2974 0         FOF_GList    633.7837   19494.8732  fof_fof()/fof.c/337
  40 2975 0             Group     11.5090   19506.3822  fof_fof()/fof.c/380
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  87 2982 0         Dist2list    126.7191   26994.1118  subfind_density()/subfind_density.c/79
  87 2983 0    DataIndexTable     21.8182   27015.9300  subfind_density()/subfind_density.c/87
  87 2984 0      DataNodeList     58.1818   27074.1118  subfind_density()/subfind_density.c/89
  87 2985 0              Left    126.7191   27200.8309  subfind_density()/subfind_density.c/91
  87 2986 0             Right    126.7191   27327.5501  subfind_density()/subfind_density.c/92
  40 2976 0   DomainNodeIndex      0.3473   19506.7295  force_treeallocate()/forcetree.c/4800
  48 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  48 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  48 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  48 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  48 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  48 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  48 2973 0         FOF_PList    380.1103   18860.9296  fof_fof()/fof.c/249
  48 2974 0         FOF_GList    633.5172   19494.4467  fof_fof()/fof.c/337
  48 2975 0             Group     13.7710   19508.2177  fof_fof()/fof.c/380
  48 2976 0   DomainNodeIndex      0.3473   19508.5650  force_treeallocate()/forcetree.c/4800
  48 2977 0        Nodes_base   3737.5399   23246.1049  force_treeallocate()/forcetree.c/4803
  87 2987 0              Todo     15.8399   27343.3900  subfind_density()/subfind_density.c/93
  87 2988 0             DM_Vx    126.7191   27470.1091  subfind_density()/subfind_density.c/95
  87 2989 0             DM_Vy    126.7191   27596.8283  subfind_density()/subfind_density.c/96
  40 2977 0        Nodes_base   3737.5399   23244.2694  force_treeallocate()/forcetree.c/4803
  40 2978 0     Extnodes_base   3114.6165   26358.8859  force_treeallocate()/forcetree.c/4811
  40 2979 0          Nextnode    222.3025   26581.1884  force_treeallocate()/forcetree.c/4820
  48 2978 0     Extnodes_base   3114.6165   26360.7215  force_treeallocate()/forcetree.c/4811
  48 2979 0          Nextnode    222.3025   26583.0240  force_treeallocate()/forcetree.c/4820
  87 2990 0             DM_Vz    126.7191   27723.5474  subfind_density()/subfind_density.c/97
  87 2991 0    	  DensDataGet    131.9354   27855.4828  subfind_density()/subfind_density.c/161
  87 2992 0  	  DensDataResult     98.9516   27954.4344  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  40 2980 0            Father    221.9056   26803.0940  force_treeallocate()/forcetree.c/4827
  40 2981 0           Ngblist     63.3784   26866.4724  subfind_density()/subfind_density.c/78
  48 2980 0            Father    221.9056   26804.9295  force_treeallocate()/forcetree.c/4827
  48 2981 0           Ngblist     63.3517   26868.2813  subfind_density()/subfind_density.c/78
  48 2982 0         Dist2list    126.7034   26994.9847  subfind_density()/subfind_density.c/79
  48 2983 0    DataIndexTable     21.8182   27016.8029  subfind_density()/subfind_density.c/87

Task=87: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.5656 MB).
task 87: endrun called with an error level of 812


  40 2982 0         Dist2list    126.7567   26993.2291  subfind_density()/subfind_density.c/79
  40 2983 0    DataIndexTable     21.8182   27015.0473  subfind_density()/subfind_density.c/87
  48 2984 0      DataNodeList     58.1818   27074.9847  subfind_density()/subfind_density.c/89
  48 2985 0              Left    126.7034   27201.6881  subfind_density()/subfind_density.c/91
  48 2986 0             Right    126.7034   27328.3915  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 87
  40 2984 0      DataNodeList     58.1818   27073.2291  subfind_density()/subfind_density.c/89
  48 2987 0              Todo     15.8379   27344.2295  subfind_density()/subfind_density.c/93
  48 2988 0             DM_Vx    126.7034   27470.9329  subfind_density()/subfind_density.c/95
  48 2989 0             DM_Vy    126.7034   27597.6364  subfind_density()/subfind_density.c/96
  40 2985 0              Left    126.7567   27199.9859  subfind_density()/subfind_density.c/91
  48 2990 0             DM_Vz    126.7034   27724.3398  subfind_density()/subfind_density.c/97
  40 2986 0             Right    126.7567   27326.7426  subfind_density()/subfind_density.c/92
  48 2991 0    	  DensDataGet    126.7787   27851.1185  subfind_density()/subfind_density.c/161
  40 2987 0              Todo     15.8446   27342.5872  subfind_density()/subfind_density.c/93
  48 2992 0  	  DensDataResult     95.0840   27946.2025  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  40 2988 0             DM_Vx    126.7567   27469.3439  subfind_density()/subfind_density.c/95

Task=48: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=53.7975 MB).
  40 2989 0             DM_Vy    126.7567   27596.1007  subfind_density()/subfind_density.c/96
  40 2990 0             DM_Vz    126.7567   27722.8574  subfind_density()/subfind_density.c/97
task 48: endrun called with an error level of 812


  40 2991 0    	  DensDataGet    132.5172   27855.3746  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 48
  40 2992 0  	  DensDataResult     99.3879   27954.7625  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 48:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value

Task=40: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.2375 MB).
task 40: endrun called with an error level of 812


task 48: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 40
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 48
task 40: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 40:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 40
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  81 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  81 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  81 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  81 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  81 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  81 2973 0         FOF_PList    380.0433   18860.8625  fof_fof()/fof.c/249
  81 2974 0         FOF_GList    633.4055   19494.2680  fof_fof()/fof.c/337
  81 2975 0             Group     10.2587   19504.5268  fof_fof()/fof.c/380
  81 2976 0   DomainNodeIndex      0.3473   19504.8741  force_treeallocate()/forcetree.c/4800
  81 2977 0        Nodes_base   3737.5399   23242.4139  force_treeallocate()/forcetree.c/4803
  81 2978 0     Extnodes_base   3114.6165   26357.0305  force_treeallocate()/forcetree.c/4811
  81 2979 0          Nextnode    222.3025   26579.3330  force_treeallocate()/forcetree.c/4820
  81 2980 0            Father    221.9056   26801.2386  force_treeallocate()/forcetree.c/4827
  81 2981 0           Ngblist     63.3406   26864.5791  subfind_density()/subfind_density.c/78
  81 2982 0         Dist2list    126.6811   26991.2602  subfind_density()/subfind_density.c/79
  81 2983 0    DataIndexTable     21.8182   27013.0784  subfind_density()/subfind_density.c/87
  81 2984 0      DataNodeList     58.1818   27071.2602  subfind_density()/subfind_density.c/89
  81 2985 0              Left    126.6811   27197.9413  subfind_density()/subfind_density.c/91
  81 2986 0             Right    126.6811   27324.6224  subfind_density()/subfind_density.c/92
  81 2987 0              Todo     15.8351   27340.4575  subfind_density()/subfind_density.c/93
  81 2988 0             DM_Vx    126.6811   27467.1386  subfind_density()/subfind_density.c/95
  81 2989 0             DM_Vy    126.6811   27593.8197  subfind_density()/subfind_density.c/96
  81 2990 0             DM_Vz    126.6811   27720.5008  subfind_density()/subfind_density.c/97
  81 2991 0    	  DensDataGet    124.8007   27845.3015  subfind_density()/subfind_density.c/161
  81 2992 0  	  DensDataResult     93.6005   27938.9020  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=81: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.098 MB).
task 81: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 81
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  80 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  80 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  80 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  80 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  80 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  47 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  47 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  47 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  47 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  47 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  47 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  47 2973 0         FOF_PList    380.5670   18861.3862  fof_fof()/fof.c/249
  47 2974 0         FOF_GList    634.2783   19495.6646  fof_fof()/fof.c/337
  47 2975 0             Group     11.5817   19507.2462  fof_fof()/fof.c/380
  47 2976 0   DomainNodeIndex      0.3473   19507.5936  force_treeallocate()/forcetree.c/4800
  47 2977 0        Nodes_base   3737.5399   23245.1334  force_treeallocate()/forcetree.c/4803
  47 2978 0     Extnodes_base   3114.6165   26359.7500  force_treeallocate()/forcetree.c/4811
  47 2979 0          Nextnode    222.3025   26582.0525  force_treeallocate()/forcetree.c/4820
  47 2980 0            Father    221.9056   26803.9580  force_treeallocate()/forcetree.c/4827
  47 2981 0           Ngblist     63.4278   26867.3859  subfind_density()/subfind_density.c/78
  47 2982 0         Dist2list    126.8557   26994.2415  subfind_density()/subfind_density.c/79
  80 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  80 2973 0         FOF_PList    380.5070   18861.3263  fof_fof()/fof.c/249
  80 2974 0         FOF_GList    634.1784   19495.5047  fof_fof()/fof.c/337
  80 2975 0             Group     11.7207   19507.2254  fof_fof()/fof.c/380
  80 2976 0   DomainNodeIndex      0.3473   19507.5727  force_treeallocate()/forcetree.c/4800
  80 2977 0        Nodes_base   3737.5399   23245.1125  force_treeallocate()/forcetree.c/4803
  80 2978 0     Extnodes_base   3114.6165   26359.7291  force_treeallocate()/forcetree.c/4811
  80 2979 0          Nextnode    222.3025   26582.0316  force_treeallocate()/forcetree.c/4820
  80 2980 0            Father    221.9056   26803.9372  force_treeallocate()/forcetree.c/4827
  47 2983 0    DataIndexTable     21.8182   27016.0597  subfind_density()/subfind_density.c/87
  47 2984 0      DataNodeList     58.1818   27074.2415  subfind_density()/subfind_density.c/89
  47 2985 0              Left    126.8557   27201.0972  subfind_density()/subfind_density.c/91
  47 2986 0             Right    126.8557   27327.9529  subfind_density()/subfind_density.c/92
  47 2987 0              Todo     15.8570   27343.8098  subfind_density()/subfind_density.c/93
  80 2981 0           Ngblist     63.4178   26867.3550  subfind_density()/subfind_density.c/78
  80 2982 0         Dist2list    126.8357   26994.1907  subfind_density()/subfind_density.c/79
  80 2983 0    DataIndexTable     21.8182   27016.0089  subfind_density()/subfind_density.c/87
  47 2988 0             DM_Vx    126.8557   27470.6655  subfind_density()/subfind_density.c/95
  47 2989 0             DM_Vy    126.8557   27597.5211  subfind_density()/subfind_density.c/96
  47 2990 0             DM_Vz    126.8557   27724.3768  subfind_density()/subfind_density.c/97
  80 2984 0      DataNodeList     58.1818   27074.1907  subfind_density()/subfind_density.c/89
  80 2985 0              Left    126.8357   27201.0263  subfind_density()/subfind_density.c/91
  80 2986 0             Right    126.8357   27327.8620  subfind_density()/subfind_density.c/92
  47 2991 0    	  DensDataGet    117.7951   27842.1719  subfind_density()/subfind_density.c/161
  47 2992 0  	  DensDataResult     88.3463   27930.5182  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  80 2987 0              Todo     15.8545   27343.7165  subfind_density()/subfind_density.c/93
  80 2988 0             DM_Vx    126.8357   27470.5522  subfind_density()/subfind_density.c/95
  80 2989 0             DM_Vy    126.8357   27597.3878  subfind_density()/subfind_density.c/96

Task=47: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.4818 MB).
task 47: endrun called with an error level of 812


  80 2990 0             DM_Vz    126.8357   27724.2235  subfind_density()/subfind_density.c/97
  80 2991 0    	  DensDataGet    121.5245   27845.7480  subfind_density()/subfind_density.c/161
  80 2992 0  	  DensDataResult     91.1434   27936.8914  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 47

Task=80: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=63.1086 MB).
task 80: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 80
task 80: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 80:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 80
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  78 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  78 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  78 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  78 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  78 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  78 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  78 2973 0         FOF_PList    380.3781   18861.1973  fof_fof()/fof.c/249
  78 2974 0         FOF_GList    633.9635   19495.1609  fof_fof()/fof.c/337
  49 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  49 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  49 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  49 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  49 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  49 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  49 2973 0         FOF_PList    379.6819   18860.5012  fof_fof()/fof.c/249
  49 2974 0         FOF_GList    632.8032   19493.3044  fof_fof()/fof.c/337
  49 2975 0             Group     11.2289   19504.5332  fof_fof()/fof.c/380
  49 2976 0   DomainNodeIndex      0.3473   19504.8805  force_treeallocate()/forcetree.c/4800
  49 2977 0        Nodes_base   3737.5399   23242.4204  force_treeallocate()/forcetree.c/4803
  49 2978 0     Extnodes_base   3114.6165   26357.0369  force_treeallocate()/forcetree.c/4811
  78 2975 0             Group     11.6543   19506.8152  fof_fof()/fof.c/380
  78 2976 0   DomainNodeIndex      0.3473   19507.1625  force_treeallocate()/forcetree.c/4800
  49 2979 0          Nextnode    222.3025   26579.3394  force_treeallocate()/forcetree.c/4820
  49 2980 0            Father    221.9056   26801.2450  force_treeallocate()/forcetree.c/4827
  49 2981 0           Ngblist     63.2803   26864.5254  subfind_density()/subfind_density.c/78
  49 2982 0         Dist2list    126.5606   26991.0860  subfind_density()/subfind_density.c/79
  49 2983 0    DataIndexTable     21.8182   27012.9042  subfind_density()/subfind_density.c/87
  78 2977 0        Nodes_base   3737.5399   23244.7023  force_treeallocate()/forcetree.c/4803
  78 2978 0     Extnodes_base   3114.6165   26359.3189  force_treeallocate()/forcetree.c/4811
  78 2979 0          Nextnode    222.3025   26581.6214  force_treeallocate()/forcetree.c/4820
  49 2984 0      DataNodeList     58.1818   27071.0860  subfind_density()/subfind_density.c/89
  49 2985 0              Left    126.5606   27197.6466  subfind_density()/subfind_density.c/91
  49 2986 0             Right    126.5606   27324.2072  subfind_density()/subfind_density.c/92
  78 2980 0            Father    221.9056   26803.5270  force_treeallocate()/forcetree.c/4827
  78 2981 0           Ngblist     63.3964   26866.9233  subfind_density()/subfind_density.c/78
  78 2982 0         Dist2list    126.7927   26993.7160  subfind_density()/subfind_density.c/79
  49 2987 0              Todo     15.8201   27340.0273  subfind_density()/subfind_density.c/93
  49 2988 0             DM_Vx    126.5606   27466.5880  subfind_density()/subfind_density.c/95
  49 2989 0             DM_Vy    126.5606   27593.1486  subfind_density()/subfind_density.c/96
  78 2983 0    DataIndexTable     21.8182   27015.5342  subfind_density()/subfind_density.c/87
  78 2984 0      DataNodeList     58.1818   27073.7160  subfind_density()/subfind_density.c/89
  78 2985 0              Left    126.7927   27200.5087  subfind_density()/subfind_density.c/91
  49 2990 0             DM_Vz    126.5606   27719.7092  subfind_density()/subfind_density.c/97
  49 2991 0    	  DensDataGet    129.8744   27849.5836  subfind_density()/subfind_density.c/161
  49 2992 0  	  DensDataResult     97.4058   27946.9894  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  78 2986 0             Right    126.7927   27327.3014  subfind_density()/subfind_density.c/92

Task=49: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=53.0106 MB).
task 49: endrun called with an error level of 812


  38 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  38 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  38 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  38 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  38 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  38 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  38 2973 0         FOF_PList    380.4877   18861.3069  fof_fof()/fof.c/249
  38 2974 0         FOF_GList    634.1462   19495.4531  fof_fof()/fof.c/337
  38 2975 0             Group     10.2587   19505.7118  fof_fof()/fof.c/380
  38 2976 0   DomainNodeIndex      0.3473   19506.0591  force_treeallocate()/forcetree.c/4800
  38 2977 0        Nodes_base   3737.5399   23243.5990  force_treeallocate()/forcetree.c/4803
  38 2978 0     Extnodes_base   3114.6165   26358.2155  force_treeallocate()/forcetree.c/4811
  89 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  89 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  89 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  89 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  89 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  89 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  89 2973 0         FOF_PList    380.5952   18861.4145  fof_fof()/fof.c/249
  89 2974 0         FOF_GList    634.3254   19495.7399  fof_fof()/fof.c/337
  89 2975 0             Group     11.3648   19507.1047  fof_fof()/fof.c/380
  89 2976 0   DomainNodeIndex      0.3473   19507.4520  force_treeallocate()/forcetree.c/4800
  89 2977 0        Nodes_base   3737.5399   23244.9919  force_treeallocate()/forcetree.c/4803
  89 2978 0     Extnodes_base   3114.6165   26359.6084  force_treeallocate()/forcetree.c/4811
  89 2979 0          Nextnode    222.3025   26581.9109  force_treeallocate()/forcetree.c/4820
  89 2980 0            Father    221.9056   26803.8165  force_treeallocate()/forcetree.c/4827
  89 2981 0           Ngblist     63.4325   26867.2490  subfind_density()/subfind_density.c/78
  78 2987 0              Todo     15.8491   27343.1505  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 49
  38 2979 0          Nextnode    222.3025   26580.5180  force_treeallocate()/forcetree.c/4820
  38 2980 0            Father    221.9056   26802.4236  force_treeallocate()/forcetree.c/4827
  89 2982 0         Dist2list    126.8651   26994.1141  subfind_density()/subfind_density.c/79
  89 2983 0    DataIndexTable     21.8182   27015.9323  subfind_density()/subfind_density.c/87
  89 2984 0      DataNodeList     58.1818   27074.1141  subfind_density()/subfind_density.c/89
  89 2985 0              Left    126.8651   27200.9792  subfind_density()/subfind_density.c/91
  89 2986 0             Right    126.8651   27327.8443  subfind_density()/subfind_density.c/92
  78 2988 0             DM_Vx    126.7927   27469.9432  subfind_density()/subfind_density.c/95
  38 2981 0           Ngblist     63.4146   26865.8382  subfind_density()/subfind_density.c/78
  38 2982 0         Dist2list    126.8292   26992.6675  subfind_density()/subfind_density.c/79
  38 2983 0    DataIndexTable     21.8182   27014.4856  subfind_density()/subfind_density.c/87
  38 2984 0      DataNodeList     58.1818   27072.6674  subfind_density()/subfind_density.c/89
  89 2987 0              Todo     15.8581   27343.7024  subfind_density()/subfind_density.c/93
  89 2988 0             DM_Vx    126.8651   27470.5675  subfind_density()/subfind_density.c/95
  89 2989 0             DM_Vy    126.8651   27597.4326  subfind_density()/subfind_density.c/96
  89 2990 0             DM_Vz    126.8651   27724.2976  subfind_density()/subfind_density.c/97
  78 2989 0             DM_Vy    126.7927   27596.7359  subfind_density()/subfind_density.c/96
  38 2985 0              Left    126.8292   27199.4967  subfind_density()/subfind_density.c/91
  38 2986 0             Right    126.8292   27326.3259  subfind_density()/subfind_density.c/92
  38 2987 0              Todo     15.8537   27342.1796  subfind_density()/subfind_density.c/93
  89 2991 0    	  DensDataGet    145.9551   27870.2528  subfind_density()/subfind_density.c/161
  89 2992 0  	  DensDataResult    109.4664   27979.7191  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=89: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=20.2809 MB).
  78 2990 0             DM_Vz    126.7927   27723.5286  subfind_density()/subfind_density.c/97
  38 2988 0             DM_Vx    126.8292   27469.0088  subfind_density()/subfind_density.c/95
  38 2989 0             DM_Vy    126.8292   27595.8380  subfind_density()/subfind_density.c/96
task 89: endrun called with an error level of 812


  78 2991 0    	  DensDataGet    118.0079   27841.5365  subfind_density()/subfind_density.c/161
  38 2990 0             DM_Vz    126.8292   27722.6673  subfind_density()/subfind_density.c/97
  38 2991 0    	  DensDataGet    111.5852   27834.2525  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 89
  78 2992 0  	  DensDataResult     88.5059   27930.0424  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  38 2992 0  	  DensDataResult     83.6889   27917.9414  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=38: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.0586 MB).
task 38: endrun called with an error level of 812



application called MPI_Abort(MPI_COMM_WORLD, 812) - process 38

Task=78: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.9576 MB).
task 78: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  44 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  44 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  44 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  44 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  44 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  44 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  44 2973 0         FOF_PList    380.0045   18860.8238  fof_fof()/fof.c/249
  44 2974 0         FOF_GList    633.3409   19494.1647  fof_fof()/fof.c/337
  44 2975 0             Group     10.2587   19504.4234  fof_fof()/fof.c/380
  44 2976 0   DomainNodeIndex      0.3473   19504.7707  force_treeallocate()/forcetree.c/4800
  44 2977 0        Nodes_base   3737.5399   23242.3106  force_treeallocate()/forcetree.c/4803
  44 2978 0     Extnodes_base   3114.6165   26356.9271  force_treeallocate()/forcetree.c/4811
  83 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  83 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  83 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  83 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  83 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  83 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  83 2973 0         FOF_PList    380.5946   18861.4138  fof_fof()/fof.c/249
  83 2974 0         FOF_GList    634.3243   19495.7381  fof_fof()/fof.c/337
  83 2975 0             Group     11.1604   19506.8985  fof_fof()/fof.c/380
  83 2976 0   DomainNodeIndex      0.3473   19507.2458  force_treeallocate()/forcetree.c/4800
  83 2977 0        Nodes_base   3737.5399   23244.7857  force_treeallocate()/forcetree.c/4803
  83 2978 0     Extnodes_base   3114.6165   26359.4022  force_treeallocate()/forcetree.c/4811
  83 2979 0          Nextnode    222.3025   26581.7047  force_treeallocate()/forcetree.c/4820
  44 2979 0          Nextnode    222.3025   26579.2296  force_treeallocate()/forcetree.c/4820
  44 2980 0            Father    221.9056   26801.1352  force_treeallocate()/forcetree.c/4827
  44 2981 0           Ngblist     63.3341   26864.4693  subfind_density()/subfind_density.c/78
  44 2982 0         Dist2list    126.6682   26991.1375  subfind_density()/subfind_density.c/79
  83 2980 0            Father    221.9056   26803.6103  force_treeallocate()/forcetree.c/4827
  83 2981 0           Ngblist     63.4324   26867.0428  subfind_density()/subfind_density.c/78
  83 2982 0         Dist2list    126.8649   26993.9076  subfind_density()/subfind_density.c/79
  83 2983 0    DataIndexTable     21.8182   27015.7258  subfind_density()/subfind_density.c/87
  44 2983 0    DataIndexTable     21.8182   27012.9557  subfind_density()/subfind_density.c/87
  44 2984 0      DataNodeList     58.1818   27071.1375  subfind_density()/subfind_density.c/89
  83 2984 0      DataNodeList     58.1818   27073.9076  subfind_density()/subfind_density.c/89
  83 2985 0              Left    126.8649   27200.7725  subfind_density()/subfind_density.c/91
  83 2986 0             Right    126.8649   27327.6373  subfind_density()/subfind_density.c/92
  83 2987 0              Todo     15.8581   27343.4954  subfind_density()/subfind_density.c/93
  44 2985 0              Left    126.6682   27197.8057  subfind_density()/subfind_density.c/91
  44 2986 0             Right    126.6682   27324.4738  subfind_density()/subfind_density.c/92
  44 2987 0              Todo     15.8335   27340.3074  subfind_density()/subfind_density.c/93
  44 2988 0             DM_Vx    126.6682   27466.9755  subfind_density()/subfind_density.c/95
  83 2988 0             DM_Vx    126.8649   27470.3603  subfind_density()/subfind_density.c/95
  83 2989 0             DM_Vy    126.8649   27597.2251  subfind_density()/subfind_density.c/96
  83 2990 0             DM_Vz    126.8649   27724.0900  subfind_density()/subfind_density.c/97
  44 2989 0             DM_Vy    126.6682   27593.6437  subfind_density()/subfind_density.c/96
  44 2990 0             DM_Vz    126.6682   27720.3119  subfind_density()/subfind_density.c/97
  44 2991 0    	  DensDataGet    133.6855   27853.9974  subfind_density()/subfind_density.c/161
  83 2991 0    	  DensDataGet    124.0646   27848.1546  subfind_density()/subfind_density.c/161
  83 2992 0  	  DensDataResult     93.0484   27941.2030  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  44 2992 0  	  DensDataResult    100.2641   27954.2615  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=83: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.797 MB).
task 83: endrun called with an error level of 812



Task=44: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.7385 MB).
task 44: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 83
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 44
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  91 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  91 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  91 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  91 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  91 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  91 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  91 2973 0         FOF_PList    380.4069   18861.2262  fof_fof()/fof.c/249
  91 2974 0         FOF_GList    634.0116   19495.2378  fof_fof()/fof.c/337
  91 2975 0             Group     11.1417   19506.3795  fof_fof()/fof.c/380
  91 2976 0   DomainNodeIndex      0.3473   19506.7268  force_treeallocate()/forcetree.c/4800
  91 2977 0        Nodes_base   3737.5399   23244.2666  force_treeallocate()/forcetree.c/4803
  91 2978 0     Extnodes_base   3114.6165   26358.8832  force_treeallocate()/forcetree.c/4811
  91 2979 0          Nextnode    222.3025   26581.1857  force_treeallocate()/forcetree.c/4820
  91 2980 0            Father    221.9056   26803.0913  force_treeallocate()/forcetree.c/4827
  91 2981 0           Ngblist     63.4012   26866.4924  subfind_density()/subfind_density.c/78
  91 2982 0         Dist2list    126.8023   26993.2948  subfind_density()/subfind_density.c/79
  91 2983 0    DataIndexTable     21.8182   27015.1129  subfind_density()/subfind_density.c/87
  36 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  36 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  36 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  36 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  36 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  36 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  36 2973 0         FOF_PList    380.6048   18861.4240  fof_fof()/fof.c/249
  36 2974 0         FOF_GList    634.3413   19495.7653  fof_fof()/fof.c/337
  36 2975 0             Group     13.5656   19509.3308  fof_fof()/fof.c/380
  36 2976 0   DomainNodeIndex      0.3473   19509.6781  force_treeallocate()/forcetree.c/4800
  36 2977 0        Nodes_base   3737.5399   23247.2180  force_treeallocate()/forcetree.c/4803
  91 2984 0      DataNodeList     58.1818   27073.2947  subfind_density()/subfind_density.c/89
  91 2985 0              Left    126.8023   27200.0970  subfind_density()/subfind_density.c/91
  91 2986 0             Right    126.8023   27326.8994  subfind_density()/subfind_density.c/92
  91 2987 0              Todo     15.8503   27342.7497  subfind_density()/subfind_density.c/93
  91 2988 0             DM_Vx    126.8023   27469.5520  subfind_density()/subfind_density.c/95
  91 2989 0             DM_Vy    126.8023   27596.3543  subfind_density()/subfind_density.c/96
  91 2990 0             DM_Vz    126.8023   27723.1566  subfind_density()/subfind_density.c/97
  36 2978 0     Extnodes_base   3114.6165   26361.8345  force_treeallocate()/forcetree.c/4811
  36 2979 0          Nextnode    222.3025   26584.1371  force_treeallocate()/forcetree.c/4820
  36 2980 0            Father    221.9056   26806.0426  force_treeallocate()/forcetree.c/4827
  36 2981 0           Ngblist     63.4341   26869.4768  subfind_density()/subfind_density.c/78
  36 2982 0         Dist2list    126.8683   26996.3450  subfind_density()/subfind_density.c/79
  91 2991 0    	  DensDataGet    111.3177   27834.4743  subfind_density()/subfind_density.c/161
  91 2992 0  	  DensDataResult     83.4883   27917.9627  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  36 2983 0    DataIndexTable     21.8182   27018.1632  subfind_density()/subfind_density.c/87
  36 2984 0      DataNodeList     58.1818   27076.3450  subfind_density()/subfind_density.c/89
  36 2985 0              Left    126.8683   27203.2132  subfind_density()/subfind_density.c/91

Task=91: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.0373 MB).
task 91: endrun called with an error level of 812


  36 2986 0             Right    126.8683   27330.0815  subfind_density()/subfind_density.c/92
  36 2987 0              Todo     15.8585   27345.9400  subfind_density()/subfind_density.c/93
  36 2988 0             DM_Vx    126.8683   27472.8083  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 91
  36 2989 0             DM_Vy    126.8683   27599.6766  subfind_density()/subfind_density.c/96
  36 2990 0             DM_Vz    126.8683   27726.5448  subfind_density()/subfind_density.c/97
  36 2991 0    	  DensDataGet    121.0875   27847.6323  subfind_density()/subfind_density.c/161
  36 2992 0  	  DensDataResult     90.8156   27938.4479  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=36: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.5521 MB).
task 36: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 36
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  39 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  39 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  39 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  39 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  39 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  39 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  39 2973 0         FOF_PList    380.5679   18861.3871  fof_fof()/fof.c/249
  39 2974 0         FOF_GList    634.2798   19495.6669  fof_fof()/fof.c/337
  39 2975 0             Group     11.8618   19507.5287  fof_fof()/fof.c/380
  39 2976 0   DomainNodeIndex      0.3473   19507.8760  force_treeallocate()/forcetree.c/4800
  88 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  88 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  88 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  88 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  88 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  88 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  39 2977 0        Nodes_base   3737.5399   23245.4159  force_treeallocate()/forcetree.c/4803
  39 2978 0     Extnodes_base   3114.6165   26360.0324  force_treeallocate()/forcetree.c/4811
  39 2979 0          Nextnode    222.3025   26582.3349  force_treeallocate()/forcetree.c/4820
  39 2980 0            Father    221.9056   26804.2405  force_treeallocate()/forcetree.c/4827
  88 2973 0         FOF_PList    380.4476   18861.2669  fof_fof()/fof.c/249
  88 2974 0         FOF_GList    634.0794   19495.3463  fof_fof()/fof.c/337
  88 2975 0             Group     11.7300   19507.0763  fof_fof()/fof.c/380
  39 2981 0           Ngblist     63.4280   26867.6685  subfind_density()/subfind_density.c/78
  39 2982 0         Dist2list    126.8560   26994.5245  subfind_density()/subfind_density.c/79
  39 2983 0    DataIndexTable     21.8182   27016.3426  subfind_density()/subfind_density.c/87
  88 2976 0   DomainNodeIndex      0.3473   19507.4236  force_treeallocate()/forcetree.c/4800
  88 2977 0        Nodes_base   3737.5399   23244.9635  force_treeallocate()/forcetree.c/4803
  39 2984 0      DataNodeList     58.1818   27074.5244  subfind_density()/subfind_density.c/89
  39 2985 0              Left    126.8560   27201.3804  subfind_density()/subfind_density.c/91
  39 2986 0             Right    126.8560   27328.2363  subfind_density()/subfind_density.c/92
  88 2978 0     Extnodes_base   3114.6165   26359.5800  force_treeallocate()/forcetree.c/4811
  88 2979 0          Nextnode    222.3025   26581.8825  force_treeallocate()/forcetree.c/4820
  88 2980 0            Father    221.9056   26803.7881  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  39 2987 0              Todo     15.8570   27344.0933  subfind_density()/subfind_density.c/93
  39 2988 0             DM_Vx    126.8560   27470.9493  subfind_density()/subfind_density.c/95
  39 2989 0             DM_Vy    126.8560   27597.8052  subfind_density()/subfind_density.c/96
  88 2981 0           Ngblist     63.4079   26867.1961  subfind_density()/subfind_density.c/78
  88 2982 0         Dist2list    126.8159   26994.0119  subfind_density()/subfind_density.c/79
  86 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  86 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  86 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  86 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  86 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  86 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  86 2973 0         FOF_PList    380.6028   18861.4221  fof_fof()/fof.c/249
  41 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  41 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  41 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  41 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  41 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  41 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  41 2973 0         FOF_PList    380.3589   18861.1781  fof_fof()/fof.c/249
  39 2990 0             DM_Vz    126.8560   27724.6612  subfind_density()/subfind_density.c/97
  39 2991 0    	  DensDataGet    123.8066   27848.4678  subfind_density()/subfind_density.c/161
  39 2992 0  	  DensDataResult     92.8549   27941.3227  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  88 2983 0    DataIndexTable     21.8182   27015.8301  subfind_density()/subfind_density.c/87
  88 2984 0      DataNodeList     58.1818   27074.0119  subfind_density()/subfind_density.c/89
  86 2974 0         FOF_GList    634.3380   19495.7601  fof_fof()/fof.c/337
  86 2975 0             Group     11.0753   19506.8354  fof_fof()/fof.c/380
  86 2976 0   DomainNodeIndex      0.3473   19507.1827  force_treeallocate()/forcetree.c/4800
  41 2974 0         FOF_GList    633.9314   19495.1095  fof_fof()/fof.c/337
  41 2975 0             Group     10.6136   19505.7231  fof_fof()/fof.c/380
  41 2976 0   DomainNodeIndex      0.3473   19506.0704  force_treeallocate()/forcetree.c/4800

Task=39: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.6773 MB).
task 39: endrun called with an error level of 812


  88 2985 0              Left    126.8159   27200.8278  subfind_density()/subfind_density.c/91
  86 2977 0        Nodes_base   3737.5399   23244.7226  force_treeallocate()/forcetree.c/4803
  86 2978 0     Extnodes_base   3114.6165   26359.3391  force_treeallocate()/forcetree.c/4811
  41 2977 0        Nodes_base   3737.5399   23243.6103  force_treeallocate()/forcetree.c/4803
  41 2978 0     Extnodes_base   3114.6165   26358.2268  force_treeallocate()/forcetree.c/4811
  41 2979 0          Nextnode    222.3025   26580.5293  force_treeallocate()/forcetree.c/4820
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 39
  88 2986 0             Right    126.8159   27327.6437  subfind_density()/subfind_density.c/92
  86 2979 0          Nextnode    222.3025   26581.6416  force_treeallocate()/forcetree.c/4820
  86 2980 0            Father    221.9056   26803.5472  force_treeallocate()/forcetree.c/4827
  86 2981 0           Ngblist     63.4338   26866.9810  subfind_density()/subfind_density.c/78
  86 2982 0         Dist2list    126.8676   26993.8486  subfind_density()/subfind_density.c/79
  41 2980 0            Father    221.9056   26802.4349  force_treeallocate()/forcetree.c/4827
  41 2981 0           Ngblist     63.3931   26865.8281  subfind_density()/subfind_density.c/78
  41 2982 0         Dist2list    126.7863   26992.6143  subfind_density()/subfind_density.c/79
  41 2983 0    DataIndexTable     21.8182   27014.4325  subfind_density()/subfind_density.c/87
  88 2987 0              Todo     15.8520   27343.4957  subfind_density()/subfind_density.c/93
  86 2983 0    DataIndexTable     21.8182   27015.6668  subfind_density()/subfind_density.c/87
  86 2984 0      DataNodeList     58.1818   27073.8486  subfind_density()/subfind_density.c/89
  86 2985 0              Left    126.8676   27200.7162  subfind_density()/subfind_density.c/91
  41 2984 0      DataNodeList     58.1818   27072.6143  subfind_density()/subfind_density.c/89
  41 2985 0              Left    126.7863   27199.4006  subfind_density()/subfind_density.c/91
  41 2986 0             Right    126.7863   27326.1869  subfind_density()/subfind_density.c/92
  88 2988 0             DM_Vx    126.8159   27470.3115  subfind_density()/subfind_density.c/95
  86 2986 0             Right    126.8676   27327.5838  subfind_density()/subfind_density.c/92
  86 2987 0              Todo     15.8585   27343.4423  subfind_density()/subfind_density.c/93
  41 2987 0              Todo     15.8483   27342.0352  subfind_density()/subfind_density.c/93
  41 2988 0             DM_Vx    126.7863   27468.8215  subfind_density()/subfind_density.c/95
  88 2989 0             DM_Vy    126.8159   27597.1274  subfind_density()/subfind_density.c/96
  86 2988 0             DM_Vx    126.8676   27470.3099  subfind_density()/subfind_density.c/95
  41 2989 0             DM_Vy    126.7863   27595.6077  subfind_density()/subfind_density.c/96
  88 2990 0             DM_Vz    126.8159   27723.9433  subfind_density()/subfind_density.c/97
  86 2989 0             DM_Vy    126.8676   27597.1775  subfind_density()/subfind_density.c/96
  41 2990 0             DM_Vz    126.7863   27722.3940  subfind_density()/subfind_density.c/97
  88 2991 0    	  DensDataGet    115.4866   27839.4299  subfind_density()/subfind_density.c/161
  86 2990 0             DM_Vz    126.8676   27724.0451  subfind_density()/subfind_density.c/97
  41 2991 0    	  DensDataGet    117.0275   27839.4215  subfind_density()/subfind_density.c/161
  88 2992 0  	  DensDataResult     86.6149   27926.0448  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  86 2991 0    	  DensDataGet    139.6986   27863.7437  subfind_density()/subfind_density.c/161
  41 2992 0  	  DensDataResult     87.7706   27927.1921  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=88: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.9552 MB).
  86 2992 0  	  DensDataResult    104.7740   27968.5177  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=41: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.8079 MB).
task 41: endrun called with an error level of 812


task 88: endrun called with an error level of 812



Task=86: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=31.4823 MB).
task 86: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 41
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 88
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 86
task 88: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 88:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 88
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  84 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  84 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  84 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  84 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  84 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  84 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  84 2973 0         FOF_PList    380.5346   18861.3539  fof_fof()/fof.c/249
  84 2974 0         FOF_GList    634.2244   19495.5782  fof_fof()/fof.c/337
  84 2975 0             Group     10.9124   19506.4906  fof_fof()/fof.c/380
  84 2976 0   DomainNodeIndex      0.3473   19506.8379  force_treeallocate()/forcetree.c/4800
  84 2977 0        Nodes_base   3737.5399   23244.3778  force_treeallocate()/forcetree.c/4803
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  84 2978 0     Extnodes_base   3114.6165   26358.9943  force_treeallocate()/forcetree.c/4811
  84 2979 0          Nextnode    222.3025   26581.2968  force_treeallocate()/forcetree.c/4820
  84 2980 0            Father    221.9056   26803.2024  force_treeallocate()/forcetree.c/4827
  84 2981 0           Ngblist     63.4224   26866.6249  subfind_density()/subfind_density.c/78
  43 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  43 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  43 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  43 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  43 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  43 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  43 2973 0         FOF_PList    380.5077   18861.3270  fof_fof()/fof.c/249
  43 2974 0         FOF_GList    634.1795   19495.5065  fof_fof()/fof.c/337
  43 2975 0             Group     12.2561   19507.7626  fof_fof()/fof.c/380
  43 2976 0   DomainNodeIndex      0.3473   19508.1099  force_treeallocate()/forcetree.c/4800
  43 2977 0        Nodes_base   3737.5399   23245.6498  force_treeallocate()/forcetree.c/4803
  43 2978 0     Extnodes_base   3114.6165   26360.2663  force_treeallocate()/forcetree.c/4811
  43 2979 0          Nextnode    222.3025   26582.5688  force_treeallocate()/forcetree.c/4820
  43 2980 0            Father    221.9056   26804.4744  force_treeallocate()/forcetree.c/4827
  43 2981 0           Ngblist     63.4180   26867.8924  subfind_density()/subfind_density.c/78
  43 2982 0         Dist2list    126.8359   26994.7283  subfind_density()/subfind_density.c/79
  43 2983 0    DataIndexTable     21.8182   27016.5464  subfind_density()/subfind_density.c/87
  84 2982 0         Dist2list    126.8449   26993.4697  subfind_density()/subfind_density.c/79
  84 2983 0    DataIndexTable     21.8182   27015.2879  subfind_density()/subfind_density.c/87
  43 2984 0      DataNodeList     58.1818   27074.7282  subfind_density()/subfind_density.c/89
  43 2985 0              Left    126.8359   27201.5641  subfind_density()/subfind_density.c/91
  43 2986 0             Right    126.8359   27328.4000  subfind_density()/subfind_density.c/92
  43 2987 0              Todo     15.8545   27344.2545  subfind_density()/subfind_density.c/93
  43 2988 0             DM_Vx    126.8359   27471.0904  subfind_density()/subfind_density.c/95
  43 2989 0             DM_Vy    126.8359   27597.9264  subfind_density()/subfind_density.c/96
  84 2984 0      DataNodeList     58.1818   27073.4697  subfind_density()/subfind_density.c/89
  84 2985 0              Left    126.8449   27200.3146  subfind_density()/subfind_density.c/91
  43 2990 0             DM_Vz    126.8359   27724.7623  subfind_density()/subfind_density.c/97
  43 2991 0    	  DensDataGet    109.8632   27834.6254  subfind_density()/subfind_density.c/161
  43 2992 0  	  DensDataResult     82.3974   27917.0228  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  84 2986 0             Right    126.8449   27327.1595  subfind_density()/subfind_density.c/92
  84 2987 0              Todo     15.8556   27343.0151  subfind_density()/subfind_density.c/93
  84 2988 0             DM_Vx    126.8449   27469.8599  subfind_density()/subfind_density.c/95

Task=43: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.9772 MB).
task 43: endrun called with an error level of 812


  84 2989 0             DM_Vy    126.8449   27596.7048  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 43
  84 2990 0             DM_Vz    126.8449   27723.5497  subfind_density()/subfind_density.c/97
  84 2991 0    	  DensDataGet    114.5161   27838.0657  subfind_density()/subfind_density.c/161
  84 2992 0  	  DensDataResult     85.8870   27923.9528  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=84: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.0472 MB).
task 84: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 84
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  90 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  90 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  90 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  90 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  90 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  90 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  90 2973 0         FOF_PList    380.5293   18861.3486  fof_fof()/fof.c/249
  90 2974 0         FOF_GList    634.2155   19495.5641  fof_fof()/fof.c/337
  90 2975 0             Group     11.2527   19506.8169  fof_fof()/fof.c/380
  90 2976 0   DomainNodeIndex      0.3473   19507.1642  force_treeallocate()/forcetree.c/4800
  90 2977 0        Nodes_base   3737.5399   23244.7040  force_treeallocate()/forcetree.c/4803
  90 2978 0     Extnodes_base   3114.6165   26359.3206  force_treeallocate()/forcetree.c/4811
  90 2979 0          Nextnode    222.3025   26581.6231  force_treeallocate()/forcetree.c/4820
  90 2980 0            Father    221.9056   26803.5287  force_treeallocate()/forcetree.c/4827
  90 2981 0           Ngblist     63.4216   26866.9502  subfind_density()/subfind_density.c/78
  90 2982 0         Dist2list    126.8431   26993.7933  subfind_density()/subfind_density.c/79
  90 2983 0    DataIndexTable     21.8182   27015.6115  subfind_density()/subfind_density.c/87
  90 2984 0      DataNodeList     58.1818   27073.7933  subfind_density()/subfind_density.c/89
  90 2985 0              Left    126.8431   27200.6364  subfind_density()/subfind_density.c/91
  90 2986 0             Right    126.8431   27327.4795  subfind_density()/subfind_density.c/92
  90 2987 0              Todo     15.8554   27343.3349  subfind_density()/subfind_density.c/93
  90 2988 0             DM_Vx    126.8431   27470.1780  subfind_density()/subfind_density.c/95
  90 2989 0             DM_Vy    126.8431   27597.0211  subfind_density()/subfind_density.c/96
  90 2990 0             DM_Vz    126.8431   27723.8642  subfind_density()/subfind_density.c/97
  90 2991 0    	  DensDataGet    108.5346   27832.3988  subfind_density()/subfind_density.c/161
  90 2992 0  	  DensDataResult     81.4010   27913.7998  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=90: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.2002 MB).
task 90: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 90
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  82 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  82 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  82 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  82 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  82 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  82 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  45 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  45 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  45 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  45 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  45 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  45 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  45 2973 0         FOF_PList    380.5995   18861.4187  fof_fof()/fof.c/249
  45 2974 0         FOF_GList    634.3325   19495.7512  fof_fof()/fof.c/337
  45 2975 0             Group     12.7791   19508.5302  fof_fof()/fof.c/380
  45 2976 0   DomainNodeIndex      0.3473   19508.8775  force_treeallocate()/forcetree.c/4800
  45 2977 0        Nodes_base   3737.5399   23246.4174  force_treeallocate()/forcetree.c/4803
  45 2978 0     Extnodes_base   3114.6165   26361.0340  force_treeallocate()/forcetree.c/4811
  45 2979 0          Nextnode    222.3025   26583.3365  force_treeallocate()/forcetree.c/4820
  45 2980 0            Father    221.9056   26805.2420  force_treeallocate()/forcetree.c/4827
  45 2981 0           Ngblist     63.4333   26868.6753  subfind_density()/subfind_density.c/78
  45 2982 0         Dist2list    126.8665   26995.5418  subfind_density()/subfind_density.c/79
  45 2983 0    DataIndexTable     21.8182   27017.3600  subfind_density()/subfind_density.c/87
  82 2973 0         FOF_PList    379.8460   18860.6653  fof_fof()/fof.c/249
  82 2974 0         FOF_GList    633.0767   19493.7420  fof_fof()/fof.c/337
  82 2975 0             Group     12.2530   19505.9950  fof_fof()/fof.c/380
  82 2976 0   DomainNodeIndex      0.3473   19506.3423  force_treeallocate()/forcetree.c/4800
  45 2984 0      DataNodeList     58.1818   27075.5418  subfind_density()/subfind_density.c/89
  45 2985 0              Left    126.8665   27202.4082  subfind_density()/subfind_density.c/91
  45 2986 0             Right    126.8665   27329.2747  subfind_density()/subfind_density.c/92
  45 2987 0              Todo     15.8583   27345.1331  subfind_density()/subfind_density.c/93
  45 2988 0             DM_Vx    126.8665   27471.9995  subfind_density()/subfind_density.c/95
  45 2989 0             DM_Vy    126.8665   27598.8660  subfind_density()/subfind_density.c/96
  45 2990 0             DM_Vz    126.8665   27725.7325  subfind_density()/subfind_density.c/97
  82 2977 0        Nodes_base   3737.5399   23243.8822  force_treeallocate()/forcetree.c/4803
  82 2978 0     Extnodes_base   3114.6165   26358.4987  force_treeallocate()/forcetree.c/4811
  45 2991 0    	  DensDataGet    117.5027   27843.2353  subfind_density()/subfind_density.c/161
  45 2992 0  	  DensDataResult     88.1271   27931.3623  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  82 2979 0          Nextnode    222.3025   26580.8012  force_treeallocate()/forcetree.c/4820
  82 2980 0            Father    221.9056   26802.7068  force_treeallocate()/forcetree.c/4827

Task=45: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.6377 MB).
task 45: endrun called with an error level of 812


  82 2981 0           Ngblist     63.3077   26866.0145  subfind_density()/subfind_density.c/78
  82 2982 0         Dist2list    126.6153   26992.6299  subfind_density()/subfind_density.c/79
  82 2983 0    DataIndexTable     21.8182   27014.4480  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 45
  82 2984 0      DataNodeList     58.1818   27072.6298  subfind_density()/subfind_density.c/89
  82 2985 0              Left    126.6153   27199.2452  subfind_density()/subfind_density.c/91
  82 2986 0             Right    126.6153   27325.8605  subfind_density()/subfind_density.c/92
  82 2987 0              Todo     15.8269   27341.6874  subfind_density()/subfind_density.c/93
  82 2988 0             DM_Vx    126.6153   27468.3028  subfind_density()/subfind_density.c/95
  82 2989 0             DM_Vy    126.6153   27594.9181  subfind_density()/subfind_density.c/96
  82 2990 0             DM_Vz    126.6153   27721.5335  subfind_density()/subfind_density.c/97
  82 2991 0    	  DensDataGet    127.8698   27849.4033  subfind_density()/subfind_density.c/161
  82 2992 0  	  DensDataResult     95.9024   27945.3057  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=82: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.6943 MB).
task 82: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 82
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  85 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  85 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  85 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  85 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  85 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  85 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  85 2973 0         FOF_PList    380.5853   18861.4046  fof_fof()/fof.c/249
  85 2974 0         FOF_GList    634.3089   19495.7134  fof_fof()/fof.c/337
  85 2975 0             Group     12.2156   19507.9290  fof_fof()/fof.c/380
  85 2976 0   DomainNodeIndex      0.3473   19508.2764  force_treeallocate()/forcetree.c/4800
  85 2977 0        Nodes_base   3737.5399   23245.8162  force_treeallocate()/forcetree.c/4803
  85 2978 0     Extnodes_base   3114.6165   26360.4328  force_treeallocate()/forcetree.c/4811
  85 2979 0          Nextnode    222.3025   26582.7353  force_treeallocate()/forcetree.c/4820
  85 2980 0            Father    221.9056   26804.6408  force_treeallocate()/forcetree.c/4827
  85 2981 0           Ngblist     63.4309   26868.0717  subfind_density()/subfind_density.c/78
  85 2982 0         Dist2list    126.8618   26994.9335  subfind_density()/subfind_density.c/79
  85 2983 0    DataIndexTable     21.8182   27016.7517  subfind_density()/subfind_density.c/87
  85 2984 0      DataNodeList     58.1818   27074.9335  subfind_density()/subfind_density.c/89
  85 2985 0              Left    126.8618   27201.7952  subfind_density()/subfind_density.c/91
  85 2986 0             Right    126.8618   27328.6570  subfind_density()/subfind_density.c/92
  85 2987 0              Todo     15.8577   27344.5147  subfind_density()/subfind_density.c/93
  85 2988 0             DM_Vx    126.8618   27471.3765  subfind_density()/subfind_density.c/95
  85 2989 0             DM_Vy    126.8618   27598.2383  subfind_density()/subfind_density.c/96
  85 2990 0             DM_Vz    126.8618   27725.1001  subfind_density()/subfind_density.c/97
  85 2991 0    	  DensDataGet    114.9512   27840.0512  subfind_density()/subfind_density.c/161
  85 2992 0  	  DensDataResult     86.2134   27926.2646  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=85: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.7354 MB).
task 85: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 85
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  52 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  52 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  52 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  52 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  52 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  52 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  52 2973 0         FOF_PList    380.6095   18861.4287  fof_fof()/fof.c/249
  52 2974 0         FOF_GList    634.3492   19495.7779  fof_fof()/fof.c/337
  52 2975 0             Group     12.2312   19508.0091  fof_fof()/fof.c/380
  52 2976 0   DomainNodeIndex      0.3473   19508.3564  force_treeallocate()/forcetree.c/4800
  52 2977 0        Nodes_base   3737.5399   23245.8963  force_treeallocate()/forcetree.c/4803
  52 2978 0     Extnodes_base   3114.6165   26360.5128  force_treeallocate()/forcetree.c/4811
  52 2979 0          Nextnode    222.3025   26582.8153  force_treeallocate()/forcetree.c/4820
  52 2980 0            Father    221.9056   26804.7209  force_treeallocate()/forcetree.c/4827
  52 2981 0           Ngblist     63.4349   26868.1558  subfind_density()/subfind_density.c/78
  52 2982 0         Dist2list    126.8698   26995.0257  subfind_density()/subfind_density.c/79
  52 2983 0    DataIndexTable     21.8182   27016.8439  subfind_density()/subfind_density.c/87
  52 2984 0      DataNodeList     58.1818   27075.0257  subfind_density()/subfind_density.c/89
  52 2985 0              Left    126.8698   27201.8955  subfind_density()/subfind_density.c/91
  52 2986 0             Right    126.8698   27328.7653  subfind_density()/subfind_density.c/92
  52 2987 0              Todo     15.8587   27344.6241  subfind_density()/subfind_density.c/93
  52 2988 0             DM_Vx    126.8698   27471.4939  subfind_density()/subfind_density.c/95
  52 2989 0             DM_Vy    126.8698   27598.3637  subfind_density()/subfind_density.c/96
  52 2990 0             DM_Vz    126.8698   27725.2336  subfind_density()/subfind_density.c/97
  52 2991 0    	  DensDataGet    125.8418   27851.0754  subfind_density()/subfind_density.c/161
  52 2992 0  	  DensDataResult     94.3813   27945.4567  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=52: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.5433 MB).
task 52: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 52
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  54 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  54 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  54 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  54 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  54 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  54 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  54 2973 0         FOF_PList    380.5002   18861.3194  fof_fof()/fof.c/249
  54 2974 0         FOF_GList    634.1670   19495.4864  fof_fof()/fof.c/337
  54 2975 0             Group     10.2587   19505.7451  fof_fof()/fof.c/380
  54 2976 0   DomainNodeIndex      0.3473   19506.0925  force_treeallocate()/forcetree.c/4800
  54 2977 0        Nodes_base   3737.5399   23243.6323  force_treeallocate()/forcetree.c/4803
  54 2978 0     Extnodes_base   3114.6165   26358.2489  force_treeallocate()/forcetree.c/4811
  54 2979 0          Nextnode    222.3025   26580.5514  force_treeallocate()/forcetree.c/4820
  54 2980 0            Father    221.9056   26802.4569  force_treeallocate()/forcetree.c/4827
  54 2981 0           Ngblist     63.4167   26865.8736  subfind_density()/subfind_density.c/78
  54 2982 0         Dist2list    126.8334   26992.7070  subfind_density()/subfind_density.c/79
  54 2983 0    DataIndexTable     21.8182   27014.5252  subfind_density()/subfind_density.c/87
  54 2984 0      DataNodeList     58.1818   27072.7070  subfind_density()/subfind_density.c/89
  54 2985 0              Left    126.8334   27199.5404  subfind_density()/subfind_density.c/91
  54 2986 0             Right    126.8334   27326.3738  subfind_density()/subfind_density.c/92
  54 2987 0              Todo     15.8542   27342.2280  subfind_density()/subfind_density.c/93
  54 2988 0             DM_Vx    126.8334   27469.0614  subfind_density()/subfind_density.c/95
  54 2989 0             DM_Vy    126.8334   27595.8948  subfind_density()/subfind_density.c/96
  54 2990 0             DM_Vz    126.8334   27722.7282  subfind_density()/subfind_density.c/97
  54 2991 0    	  DensDataGet    110.7976   27833.5258  subfind_density()/subfind_density.c/161
  54 2992 0  	  DensDataResult     83.0982   27916.6240  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=54: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.376 MB).
task 54: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 54
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  53 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  53 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  53 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  53 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  53 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  53 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  53 2973 0         FOF_PList    380.5048   18861.3241  fof_fof()/fof.c/249
  53 2974 0         FOF_GList    634.1747   19495.4988  fof_fof()/fof.c/337
  53 2975 0             Group     12.2312   19507.7300  fof_fof()/fof.c/380
  53 2976 0   DomainNodeIndex      0.3473   19508.0773  force_treeallocate()/forcetree.c/4800
  53 2977 0        Nodes_base   3737.5399   23245.6172  force_treeallocate()/forcetree.c/4803
  53 2978 0     Extnodes_base   3114.6165   26360.2337  force_treeallocate()/forcetree.c/4811
  53 2979 0          Nextnode    222.3025   26582.5362  force_treeallocate()/forcetree.c/4820
  53 2980 0            Father    221.9056   26804.4418  force_treeallocate()/forcetree.c/4827
  53 2981 0           Ngblist     63.4175   26867.8593  subfind_density()/subfind_density.c/78
  74 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  74 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  74 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  74 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  74 2972 0          TopNodes      5.1599   18480.8192  domain_allocate()/domain.c/570
  74 2973 0         FOF_PList    380.2183   18861.0375  fof_fof()/fof.c/249
  74 2974 0         FOF_GList    633.6971   19494.7346  fof_fof()/fof.c/337
  74 2975 0             Group     10.4839   19505.2185  fof_fof()/fof.c/380
  74 2976 0   DomainNodeIndex      0.3473   19505.5658  force_treeallocate()/forcetree.c/4800
  74 2977 0        Nodes_base   3737.5399   23243.1056  force_treeallocate()/forcetree.c/4803
  53 2982 0         Dist2list    126.8349   26994.6942  subfind_density()/subfind_density.c/79
  53 2983 0    DataIndexTable     21.8182   27016.5124  subfind_density()/subfind_density.c/87
  53 2984 0      DataNodeList     58.1818   27074.6942  subfind_density()/subfind_density.c/89
  53 2985 0              Left    126.8349   27201.5291  subfind_density()/subfind_density.c/91
  74 2978 0     Extnodes_base   3114.6165   26357.7222  force_treeallocate()/forcetree.c/4811
  74 2979 0          Nextnode    222.3025   26580.0247  force_treeallocate()/forcetree.c/4820
  74 2980 0            Father    221.9056   26801.9303  force_treeallocate()/forcetree.c/4827
  53 2986 0             Right    126.8349   27328.3641  subfind_density()/subfind_density.c/92
  53 2987 0              Todo     15.8544   27344.2185  subfind_density()/subfind_density.c/93
  53 2988 0             DM_Vx    126.8349   27471.0534  subfind_density()/subfind_density.c/95
  53 2989 0             DM_Vy    126.8349   27597.8884  subfind_density()/subfind_density.c/96
  74 2981 0           Ngblist     63.3697   26865.3000  subfind_density()/subfind_density.c/78
  74 2982 0         Dist2list    126.7394   26992.0394  subfind_density()/subfind_density.c/79
  74 2983 0    DataIndexTable     21.8182   27013.8576  subfind_density()/subfind_density.c/87
  53 2990 0             DM_Vz    126.8349   27724.7233  subfind_density()/subfind_density.c/97
  53 2991 0    	  DensDataGet    117.3622   27842.0855  subfind_density()/subfind_density.c/161
  74 2984 0      DataNodeList     58.1818   27072.0394  subfind_density()/subfind_density.c/89
  74 2985 0              Left    126.7394   27198.7788  subfind_density()/subfind_density.c/91
  74 2986 0             Right    126.7394   27325.5182  subfind_density()/subfind_density.c/92
  53 2992 0  	  DensDataResult     88.0216   27930.1071  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=53: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.8929 MB).
task 53: endrun called with an error level of 812


  74 2987 0              Todo     15.8424   27341.3606  subfind_density()/subfind_density.c/93
  74 2988 0             DM_Vx    126.7394   27468.1001  subfind_density()/subfind_density.c/95
  74 2989 0             DM_Vy    126.7394   27594.8395  subfind_density()/subfind_density.c/96
  74 2990 0             DM_Vz    126.7394   27721.5789  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 53
  74 2991 0    	  DensDataGet    138.2497   27859.8286  subfind_density()/subfind_density.c/161
  74 2992 0  	  DensDataResult    103.6873   27963.5159  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=74: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=36.4841 MB).
task 74: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 74
-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63112.56 Mb (on task=120), Smallest =   62842.59 Mb (on task=0), Average =   63040.15 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1689.18 Mb (on task=0), Smallest =    1419.50 Mb (on task=120), Average =    1491.92 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.11 Mb
AllocMem:	 Largest =    1689.18 Mb (on task=0), Smallest =    1419.50 Mb (on task=120), Average =    1491.92 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_026_z000p101/snap_026_z000p101.0.hdf5' on task=0 (contains 132936583 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  132687050  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      9667  (tot=     0000134208) masstab=0
Type 3 (bulge):   239866  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.921268

Setting next time for line of sight file to Time_next= 0.914371

Setting next time for snipshot file to Time_next= 0.915269

Setting next time for fof output to Time_next= 0.915269
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=11.6842 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=102409
After=103001
NTopleaves= 90126  NTopnodes=103001 (space for 465370)
gravity work-load balance=1.00039   memory-balance=1.00038   SPH work-load balance=0
iter=0 exchange of 2111067432 particles (ret=0)
domain decomposition done. (took 34.2934 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5879 MByte in top-level domain structure

Allocated 7296.18 MByte for BH-tree, and 0.343803 Mbyte for top-leaves.  (presently allocted 25777.3 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.2 Mbyte (on task=31), Smallest = 26119.7 Mbyte, Average = 26120 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  31 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  31 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  31 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  31 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  31 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  31 2972 0          TopNodes      5.1079   18480.7673  domain_allocate()/domain.c/570
  31 2973 0   DomainNodeIndex      0.3438   18481.1111  force_treeallocate()/forcetree.c/4800
  31 2974 0        Nodes_base   3737.4439   22218.5550  force_treeallocate()/forcetree.c/4803
  31 2975 0     Extnodes_base   3114.5366   25333.0916  force_treeallocate()/forcetree.c/4811
  31 2976 0          Nextnode    222.2985   25555.3901  force_treeallocate()/forcetree.c/4820
  31 2977 0            Father    221.9056   25777.2957  force_treeallocate()/forcetree.c/4827
  31 2978 0           Ngblist     63.4259   25840.7216  density()/density.c/690
  31 2979 0              Left    126.8519   25967.5735  density()/density.c/692
  31 2980 0             Right    126.8519   26094.4254  density()/density.c/693
  31 2981 0    DataIndexTable      7.0175   26101.4429  density()/density.c/738
  31 2982 0      DataNodeList     18.7134   26120.1563  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.049.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.101.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.3 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=93361
After=103697
NTopleaves= 90735  NTopnodes=103697 (space for 465370)
gravity work-load balance=1.00046   memory-balance=1.00047   SPH work-load balance=0
iter=0 exchange of 2110986961 particles (ret=0)
domain decomposition done. (took 30.4736 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5561 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.8 MB)
Tree construction.

Start linking particles (presently allocated=26537.6 MB)
links on local processor done (took 173.766 sec).
Marked=0234304309 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26861.4 MB) 
have done 0002952660 cross links (processed 0234304309, took 26.3094 sec)
have done 0001309047 cross links (processed 0196236112, took 28.3344 sec)
have done 0000950945 cross links (processed 0167915316, took 20.6866 sec)
have done 0000642694 cross links (processed 0116159586, took 18.8081 sec)
have done 0000423022 cross links (processed 0069768276, took 17.1603 sec)
have done 0000275124 cross links (processed 0044341742, took 16.4527 sec)
have done 0000173132 cross links (processed 0021106368, took 8.41973 sec)
have done 0000096137 cross links (processed 0008390834, took 9.37935 sec)
have done 0000054437 cross links (processed 0003709998, took 1.76296 sec)
have done 0000028954 cross links (processed 0001617436, took 2.36466 sec)
have done 0000013844 cross links (processed 0000461373, took 0.334726 sec)
have done 0000008275 cross links (processed 0001177975, took 2.49962 sec)
have done 0000005207 cross links (processed 0000784328, took 1.68799 sec)
have done 0000001794 cross links (processed 0000040934, took 0.437812 sec)
have done 0000000780 cross links (processed 0000031803, took 0.404238 sec)
have done 0000000389 cross links (processed 0000007511, took 0.369512 sec)
have done 0000000194 cross links (processed 0000002581, took 0.364681 sec)
have done 0000000119 cross links (processed 0000001696, took 0.363877 sec)
have done 0000000074 cross links (processed 0000001278, took 0.363839 sec)
have done 0000000048 cross links (processed 0000000800, took 0.36451 sec)
have done 0000000016 cross links (processed 0000000274, took 0.363736 sec)
have done 0000000007 cross links (processed 0000000063, took 0.3641 sec)
have done 0000000003 cross links (processed 0000000026, took 0.36098 sec)
have done 0000000000 cross links (processed 0000000007, took 0.36118 sec)
Local groups found.

group finding took = 332.702 sec
Start finding nearest dm-particle (presently allocated=26537.6 MB)
still finding nearest... (presently allocated=26804.8 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.807399 sec
compiling local group data and catalogue took = 12.2518 sec

Total number of groups with at least 32 particles: 1273286
Largest group has 140958548 particles.
Total number of particles in groups: 1308288697

group properties are now allocated.. (presently allocated=19506.7 MB)
computation of group properties took = 0.920731 sec
start global sorting of group catalogues number 26
LargestGroup = 109882.685190; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.218443 sec
Group catalogues globally sorted. took = 9.2312 sec
starting saving of group catalogue number 26
data//groups_026_z000p101
Group catalogues number 26 saved. took = 16.8147 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 12.2338 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 2.06431 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 12.4115 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  57 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  57 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  57 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  57 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  57 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  57 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  57 2973 0         FOF_PList    380.5668   18861.3686  fof_fof()/fof.c/249
  57 2974 0         FOF_GList    634.2780   19495.6466  fof_fof()/fof.c/337
  57 2975 0             Group     10.4579   19506.1046  fof_fof()/fof.c/380
  57 2976 0   DomainNodeIndex      0.3461   19506.4507  force_treeallocate()/forcetree.c/4800
  57 2977 0        Nodes_base   3737.5076   23243.9583  force_treeallocate()/forcetree.c/4803
  57 2978 0     Extnodes_base   3114.5897   26358.5480  force_treeallocate()/forcetree.c/4811
  57 2979 0          Nextnode    222.3012   26580.8492  force_treeallocate()/forcetree.c/4820
  57 2980 0            Father    221.9056   26802.7548  force_treeallocate()/forcetree.c/4827
  57 2981 0           Ngblist     63.4278   26866.1826  subfind_density()/subfind_density.c/78
  57 2982 0         Dist2list    126.8556   26993.0382  subfind_density()/subfind_density.c/79
  57 2983 0    DataIndexTable     21.8182   27014.8564  subfind_density()/subfind_density.c/87
  57 2984 0      DataNodeList     58.1818   27073.0382  subfind_density()/subfind_density.c/89
  57 2985 0              Left    126.8556   27199.8938  subfind_density()/subfind_density.c/91
  57 2986 0             Right    126.8556   27326.7494  subfind_density()/subfind_density.c/92
  57 2987 0              Todo     15.8570   27342.6063  subfind_density()/subfind_density.c/93
  57 2988 0             DM_Vx    126.8556   27469.4619  subfind_density()/subfind_density.c/95
  57 2989 0             DM_Vy    126.8556   27596.3175  subfind_density()/subfind_density.c/96
  57 2990 0             DM_Vz    126.8556   27723.1731  subfind_density()/subfind_density.c/97
  57 2991 0    	  DensDataGet    114.5808   27837.7540  subfind_density()/subfind_density.c/161
  57 2992 0  	  DensDataResult     85.9356   27923.6896  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=57: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.3104 MB).
task 57: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 57
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  80 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  80 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  80 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  80 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  80 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  80 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  80 2973 0         FOF_PList    380.0926   18860.8944  fof_fof()/fof.c/249
  80 2974 0         FOF_GList    633.4877   19494.3820  fof_fof()/fof.c/337
  80 2975 0             Group     12.1907   19506.5728  fof_fof()/fof.c/380
  80 2976 0   DomainNodeIndex      0.3461   19506.9189  force_treeallocate()/forcetree.c/4800
  80 2977 0        Nodes_base   3737.5076   23244.4265  force_treeallocate()/forcetree.c/4803
  80 2978 0     Extnodes_base   3114.5897   26359.0162  force_treeallocate()/forcetree.c/4811
  80 2979 0          Nextnode    222.3012   26581.3174  force_treeallocate()/forcetree.c/4820
  47 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  47 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  47 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  47 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  47 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  47 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  47 2973 0         FOF_PList    380.5704   18861.3722  fof_fof()/fof.c/249
  47 2974 0         FOF_GList    634.2839   19495.6561  fof_fof()/fof.c/337
  47 2975 0             Group     12.3391   19507.9952  fof_fof()/fof.c/380
  47 2976 0   DomainNodeIndex      0.3461   19508.3413  force_treeallocate()/forcetree.c/4800
  47 2977 0        Nodes_base   3737.5076   23245.8490  force_treeallocate()/forcetree.c/4803
  47 2978 0     Extnodes_base   3114.5897   26360.4387  force_treeallocate()/forcetree.c/4811
  47 2979 0          Nextnode    222.3012   26582.7398  force_treeallocate()/forcetree.c/4820
  80 2980 0            Father    221.9056   26803.2230  force_treeallocate()/forcetree.c/4827
  80 2981 0           Ngblist     63.3488   26866.5718  subfind_density()/subfind_density.c/78
  47 2980 0            Father    221.9056   26804.6454  force_treeallocate()/forcetree.c/4827
  47 2981 0           Ngblist     63.4284   26868.0738  subfind_density()/subfind_density.c/78
  47 2982 0         Dist2list    126.8568   26994.9306  subfind_density()/subfind_density.c/79
  47 2983 0    DataIndexTable     21.8182   27016.7488  subfind_density()/subfind_density.c/87
  47 2984 0      DataNodeList     58.1818   27074.9306  subfind_density()/subfind_density.c/89
  80 2982 0         Dist2list    126.6975   26993.2693  subfind_density()/subfind_density.c/79
  80 2983 0    DataIndexTable     21.8182   27015.0875  subfind_density()/subfind_density.c/87
  80 2984 0      DataNodeList     58.1818   27073.2693  subfind_density()/subfind_density.c/89
  47 2985 0              Left    126.8568   27201.7874  subfind_density()/subfind_density.c/91
  47 2986 0             Right    126.8568   27328.6441  subfind_density()/subfind_density.c/92
  47 2987 0              Todo     15.8571   27344.5012  subfind_density()/subfind_density.c/93
  80 2985 0              Left    126.6975   27199.9668  subfind_density()/subfind_density.c/91
  80 2986 0             Right    126.6975   27326.6643  subfind_density()/subfind_density.c/92
  80 2987 0              Todo     15.8372   27342.5015  subfind_density()/subfind_density.c/93
  47 2988 0             DM_Vx    126.8568   27471.3580  subfind_density()/subfind_density.c/95
  47 2989 0             DM_Vy    126.8568   27598.2148  subfind_density()/subfind_density.c/96
  47 2990 0             DM_Vz    126.8568   27725.0716  subfind_density()/subfind_density.c/97
  80 2988 0             DM_Vx    126.6975   27469.1991  subfind_density()/subfind_density.c/95
  80 2989 0             DM_Vy    126.6975   27595.8966  subfind_density()/subfind_density.c/96
  80 2990 0             DM_Vz    126.6975   27722.5941  subfind_density()/subfind_density.c/97
  47 2991 0    	  DensDataGet    122.7532   27847.8248  subfind_density()/subfind_density.c/161
  47 2992 0  	  DensDataResult     92.0649   27939.8898  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  80 2991 0    	  DensDataGet    128.9828   27851.5770  subfind_density()/subfind_density.c/161

Task=47: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.1102 MB).
task 47: endrun called with an error level of 812


  80 2992 0  	  DensDataResult     96.7371   27948.3141  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 47

Task=80: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.6859 MB).
task 80: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 80
task 80: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 80:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 80
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  56 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  56 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  56 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  56 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  56 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  56 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  56 2973 0         FOF_PList    379.7770   18860.5788  fof_fof()/fof.c/249
  56 2974 0         FOF_GList    632.9617   19493.5406  fof_fof()/fof.c/337
  56 2975 0             Group     11.5640   19505.1046  fof_fof()/fof.c/380
  71 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  71 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  71 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  71 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  71 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  71 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  71 2973 0         FOF_PList    380.2264   18861.0282  fof_fof()/fof.c/249
  71 2974 0         FOF_GList    633.7107   19494.7389  fof_fof()/fof.c/337
  71 2975 0             Group     11.5194   19506.2583  fof_fof()/fof.c/380
  71 2976 0   DomainNodeIndex      0.3461   19506.6044  force_treeallocate()/forcetree.c/4800
  71 2977 0        Nodes_base   3737.5076   23244.1120  force_treeallocate()/forcetree.c/4803
  71 2978 0     Extnodes_base   3114.5897   26358.7017  force_treeallocate()/forcetree.c/4811
  71 2979 0          Nextnode    222.3012   26581.0029  force_treeallocate()/forcetree.c/4820
  71 2980 0            Father    221.9056   26802.9085  force_treeallocate()/forcetree.c/4827
  71 2981 0           Ngblist     63.3711   26866.2795  subfind_density()/subfind_density.c/78
  56 2976 0   DomainNodeIndex      0.3461   19505.4507  force_treeallocate()/forcetree.c/4800
  56 2977 0        Nodes_base   3737.5076   23242.9583  force_treeallocate()/forcetree.c/4803
  71 2982 0         Dist2list    126.7421   26993.0217  subfind_density()/subfind_density.c/79
  71 2983 0    DataIndexTable     21.8182   27014.8399  subfind_density()/subfind_density.c/87
  71 2984 0      DataNodeList     58.1818   27073.0216  subfind_density()/subfind_density.c/89
  71 2985 0              Left    126.7421   27199.7638  subfind_density()/subfind_density.c/91
  56 2978 0     Extnodes_base   3114.5897   26357.5480  force_treeallocate()/forcetree.c/4811
  56 2979 0          Nextnode    222.3012   26579.8492  force_treeallocate()/forcetree.c/4820
  56 2980 0            Father    221.9056   26801.7548  force_treeallocate()/forcetree.c/4827
  71 2986 0             Right    126.7421   27326.5059  subfind_density()/subfind_density.c/92
  71 2987 0              Todo     15.8428   27342.3487  subfind_density()/subfind_density.c/93
  71 2988 0             DM_Vx    126.7421   27469.0908  subfind_density()/subfind_density.c/95
  56 2981 0           Ngblist     63.2962   26865.0510  subfind_density()/subfind_density.c/78
  56 2982 0         Dist2list    126.5923   26991.6433  subfind_density()/subfind_density.c/79
  56 2983 0    DataIndexTable     21.8182   27013.4615  subfind_density()/subfind_density.c/87
  71 2989 0             DM_Vy    126.7421   27595.8330  subfind_density()/subfind_density.c/96
  71 2990 0             DM_Vz    126.7421   27722.5751  subfind_density()/subfind_density.c/97
  71 2991 0    	  DensDataGet    122.8970   27845.4721  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  56 2984 0      DataNodeList     58.1818   27071.6433  subfind_density()/subfind_density.c/89
  56 2985 0              Left    126.5923   27198.2356  subfind_density()/subfind_density.c/91
  71 2992 0  	  DensDataResult     92.1727   27937.6448  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=71: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.3552 MB).
task 71: endrun called with an error level of 812


  83 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  83 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  83 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  83 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  83 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  83 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  83 2973 0         FOF_PList    380.5314   18861.3332  fof_fof()/fof.c/249
  83 2974 0         FOF_GList    634.2190   19495.5522  fof_fof()/fof.c/337
  83 2975 0             Group     11.5049   19507.0570  fof_fof()/fof.c/380
  83 2976 0   DomainNodeIndex      0.3461   19507.4032  force_treeallocate()/forcetree.c/4800
  83 2977 0        Nodes_base   3737.5076   23244.9108  force_treeallocate()/forcetree.c/4803
  83 2978 0     Extnodes_base   3114.5897   26359.5005  force_treeallocate()/forcetree.c/4811
  83 2979 0          Nextnode    222.3012   26581.8017  force_treeallocate()/forcetree.c/4820
  83 2980 0            Father    221.9056   26803.7072  force_treeallocate()/forcetree.c/4827
  83 2981 0           Ngblist     63.4219   26867.1291  subfind_density()/subfind_density.c/78
  83 2982 0         Dist2list    126.8438   26993.9729  subfind_density()/subfind_density.c/79
  83 2983 0    DataIndexTable     21.8182   27015.7911  subfind_density()/subfind_density.c/87
  56 2986 0             Right    126.5923   27324.8280  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 71
  83 2984 0      DataNodeList     58.1818   27073.9729  subfind_density()/subfind_density.c/89
  83 2985 0              Left    126.8438   27200.8167  subfind_density()/subfind_density.c/91
  83 2986 0             Right    126.8438   27327.6605  subfind_density()/subfind_density.c/92
  83 2987 0              Todo     15.8555   27343.5160  subfind_density()/subfind_density.c/93
  83 2988 0             DM_Vx    126.8438   27470.3598  subfind_density()/subfind_density.c/95
  83 2989 0             DM_Vy    126.8438   27597.2036  subfind_density()/subfind_density.c/96
  56 2987 0              Todo     15.8240   27340.6520  subfind_density()/subfind_density.c/93
  83 2990 0             DM_Vz    126.8438   27724.0474  subfind_density()/subfind_density.c/97
  83 2991 0    	  DensDataGet    119.2881   27843.3354  subfind_density()/subfind_density.c/161
  83 2992 0  	  DensDataResult     89.4661   27932.8015  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  56 2988 0             DM_Vx    126.5923   27467.2444  subfind_density()/subfind_density.c/95

Task=83: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.1985 MB).
task 83: endrun called with an error level of 812


  56 2989 0             DM_Vy    126.5923   27593.8367  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 83
  56 2990 0             DM_Vz    126.5923   27720.4290  subfind_density()/subfind_density.c/97
  56 2991 0    	  DensDataGet    119.7220   27840.1510  subfind_density()/subfind_density.c/161
  56 2992 0  	  DensDataResult     89.7915   27929.9425  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=56: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.0575 MB).
task 56: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 56
 An HDF5 error was detected, we'd better stop.
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
HDF5-DIAG: Error detected in HDF5 (1.8.12)   81 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  81 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  81 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  81 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  81 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  81 2973 0         FOF_PList    380.4489   18861.2507  fof_fof()/fof.c/249
  81 2974 0         FOF_GList    634.0816   19495.3323  fof_fof()/fof.c/337
  81 2975 0             Group     10.3220   19505.6543  fof_fof()/fof.c/380
  81 2976 0   DomainNodeIndex      0.3461   19506.0005  force_treeallocate()/forcetree.c/4800
  81 2977 0        Nodes_base   3737.5076   23243.5081  force_treeallocate()/forcetree.c/4803
  81 2978 0     Extnodes_base   3114.5897   26358.0978  force_treeallocate()/forcetree.c/4811
  81 2979 0          Nextnode    222.3012   26580.3989  force_treeallocate()/forcetree.c/4820
  46 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  46 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  46 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  46 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  46 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
MPI-process 56:
  81 2980 0            Father    221.9056   26802.3045  force_treeallocate()/forcetree.c/4827
  81 2981 0           Ngblist     63.4082   26865.7127  subfind_density()/subfind_density.c/78
  81 2982 0         Dist2list    126.8163   26992.5290  subfind_density()/subfind_density.c/79
  81 2983 0    DataIndexTable     21.8182   27014.3472  subfind_density()/subfind_density.c/87
  46 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  46 2973 0         FOF_PList    380.4882   18861.2899  fof_fof()/fof.c/249
  46 2974 0         FOF_GList    634.1469   19495.4369  fof_fof()/fof.c/337
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
  81 2984 0      DataNodeList     58.1818   27072.5290  subfind_density()/subfind_density.c/89
  81 2985 0              Left    126.8163   27199.3453  subfind_density()/subfind_density.c/91
  81 2986 0             Right    126.8163   27326.1616  subfind_density()/subfind_density.c/92
  46 2975 0             Group     11.1334   19506.5703  fof_fof()/fof.c/380
  46 2976 0   DomainNodeIndex      0.3461   19506.9164  force_treeallocate()/forcetree.c/4800
  46 2977 0        Nodes_base   3737.5076   23244.4240  force_treeallocate()/forcetree.c/4803
task 56: endrun called with an error level of 124


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  81 2987 0              Todo     15.8520   27342.0136  subfind_density()/subfind_density.c/93
  81 2988 0             DM_Vx    126.8163   27468.8300  subfind_density()/subfind_density.c/95
  81 2989 0             DM_Vy    126.8163   27595.6463  subfind_density()/subfind_density.c/96
  46 2978 0     Extnodes_base   3114.5897   26359.0137  force_treeallocate()/forcetree.c/4811
  46 2979 0          Nextnode    222.3012   26581.3149  force_treeallocate()/forcetree.c/4820
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 56
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  68 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  68 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  68 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  68 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  68 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  68 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  68 2973 0         FOF_PList    380.5782   18861.3800  fof_fof()/fof.c/249
  68 2974 0         FOF_GList    634.2970   19495.6770  fof_fof()/fof.c/337
  68 2975 0             Group     10.7796   19506.4566  fof_fof()/fof.c/380
  68 2976 0   DomainNodeIndex      0.3461   19506.8027  force_treeallocate()/forcetree.c/4800
  81 2990 0             DM_Vz    126.8163   27722.4626  subfind_density()/subfind_density.c/97
  81 2991 0    	  DensDataGet    130.5465   27853.0091  subfind_density()/subfind_density.c/161
  81 2992 0  	  DensDataResult     97.9099   27950.9190  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  46 2980 0            Father    221.9056   26803.2205  force_treeallocate()/forcetree.c/4827
  46 2981 0           Ngblist     63.4147   26866.6352  subfind_density()/subfind_density.c/78
  86 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  86 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  86 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  86 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  86 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  86 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  86 2973 0         FOF_PList    380.5595   18861.3613  fof_fof()/fof.c/249
  86 2974 0         FOF_GList    634.2659   19495.6272  fof_fof()/fof.c/337
  86 2975 0             Group     11.1812   19506.8084  fof_fof()/fof.c/380
  86 2976 0   DomainNodeIndex      0.3461   19507.1545  force_treeallocate()/forcetree.c/4800
  86 2977 0        Nodes_base   3737.5076   23244.6621  force_treeallocate()/forcetree.c/4803
  86 2978 0     Extnodes_base   3114.5897   26359.2518  force_treeallocate()/forcetree.c/4811
  59 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  59 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  59 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  59 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  59 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  59 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  59 2973 0         FOF_PList    380.4963   18861.2981  fof_fof()/fof.c/249
  59 2974 0         FOF_GList    634.1605   19495.4586  fof_fof()/fof.c/337
  59 2975 0             Group     10.9508   19506.4095  fof_fof()/fof.c/380
  59 2976 0   DomainNodeIndex      0.3461   19506.7556  force_treeallocate()/forcetree.c/4800
  59 2977 0        Nodes_base   3737.5076   23244.2632  force_treeallocate()/forcetree.c/4803
  59 2978 0     Extnodes_base   3114.5897   26358.8529  force_treeallocate()/forcetree.c/4811
  59 2979 0          Nextnode    222.3012   26581.1541  force_treeallocate()/forcetree.c/4820
  59 2980 0            Father    221.9056   26803.0597  force_treeallocate()/forcetree.c/4827
  59 2981 0           Ngblist     63.4161   26866.4757  subfind_density()/subfind_density.c/78
  59 2982 0         Dist2list    126.8321   26993.3078  subfind_density()/subfind_density.c/79
  41 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  41 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  41 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  41 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  41 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  41 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  41 2973 0         FOF_PList    380.4802   18861.2820  fof_fof()/fof.c/249
  41 2974 0         FOF_GList    634.1337   19495.4157  fof_fof()/fof.c/337
  41 2975 0             Group     12.3557   19507.7715  fof_fof()/fof.c/380
  41 2976 0   DomainNodeIndex      0.3461   19508.1176  force_treeallocate()/forcetree.c/4800
  41 2977 0        Nodes_base   3737.5076   23245.6252  force_treeallocate()/forcetree.c/4803
  41 2978 0     Extnodes_base   3114.5897   26360.2149  force_treeallocate()/forcetree.c/4811
  41 2979 0          Nextnode    222.3012   26582.5161  force_treeallocate()/forcetree.c/4820
  41 2980 0            Father    221.9056   26804.4217  force_treeallocate()/forcetree.c/4827
  41 2981 0           Ngblist     63.4134   26867.8350  subfind_density()/subfind_density.c/78
  68 2977 0        Nodes_base   3737.5076   23244.3103  force_treeallocate()/forcetree.c/4803
  68 2978 0     Extnodes_base   3114.5897   26358.9000  force_treeallocate()/forcetree.c/4811
  68 2979 0          Nextnode    222.3012   26581.2012  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=81: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.081 MB).
task 81: endrun called with an error level of 812


  46 2982 0         Dist2list    126.8294   26993.4646  subfind_density()/subfind_density.c/79
  46 2983 0    DataIndexTable     21.8182   27015.2827  subfind_density()/subfind_density.c/87
  46 2984 0      DataNodeList     58.1818   27073.4645  subfind_density()/subfind_density.c/89
  46 2985 0              Left    126.8294   27200.2939  subfind_density()/subfind_density.c/91
  86 2979 0          Nextnode    222.3012   26581.5530  force_treeallocate()/forcetree.c/4820
  86 2980 0            Father    221.9056   26803.4586  force_treeallocate()/forcetree.c/4827
  59 2983 0    DataIndexTable     21.8182   27015.1260  subfind_density()/subfind_density.c/87
  59 2984 0      DataNodeList     58.1818   27073.3078  subfind_density()/subfind_density.c/89
  59 2985 0              Left    126.8321   27200.1399  subfind_density()/subfind_density.c/91
  59 2986 0             Right    126.8321   27326.9720  subfind_density()/subfind_density.c/92
  59 2987 0              Todo     15.8540   27342.8260  subfind_density()/subfind_density.c/93
  41 2982 0         Dist2list    126.8267   26994.6618  subfind_density()/subfind_density.c/79
  41 2983 0    DataIndexTable     21.8182   27016.4799  subfind_density()/subfind_density.c/87
  41 2984 0      DataNodeList     58.1818   27074.6617  subfind_density()/subfind_density.c/89
  68 2980 0            Father    221.9056   26803.1068  force_treeallocate()/forcetree.c/4827
  68 2981 0           Ngblist     63.4297   26866.5365  subfind_density()/subfind_density.c/78
  68 2982 0         Dist2list    126.8594   26993.3959  subfind_density()/subfind_density.c/79
  67 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  67 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  67 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  67 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  67 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  67 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  67 2973 0         FOF_PList    380.5768   18861.3785  fof_fof()/fof.c/249
  67 2974 0         FOF_GList    634.2946   19495.6731  fof_fof()/fof.c/337
  67 2975 0             Group     10.3220   19505.9951  fof_fof()/fof.c/380
  67 2976 0   DomainNodeIndex      0.3461   19506.3413  force_treeallocate()/forcetree.c/4800
  67 2977 0        Nodes_base   3737.5076   23243.8489  force_treeallocate()/forcetree.c/4803
  67 2978 0     Extnodes_base   3114.5897   26358.4386  force_treeallocate()/forcetree.c/4811
  67 2979 0          Nextnode    222.3012   26580.7398  force_treeallocate()/forcetree.c/4820
  67 2980 0            Father    221.9056   26802.6453  force_treeallocate()/forcetree.c/4827
  67 2981 0           Ngblist     63.4295   26866.0748  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 81
  46 2986 0             Right    126.8294   27327.1233  subfind_density()/subfind_density.c/92
  86 2981 0           Ngblist     63.4266   26866.8852  subfind_density()/subfind_density.c/78
  86 2982 0         Dist2list    126.8532   26993.7383  subfind_density()/subfind_density.c/79
  86 2983 0    DataIndexTable     21.8182   27015.5565  subfind_density()/subfind_density.c/87
  59 2988 0             DM_Vx    126.8321   27469.6581  subfind_density()/subfind_density.c/95
  59 2989 0             DM_Vy    126.8321   27596.4902  subfind_density()/subfind_density.c/96
  59 2990 0             DM_Vz    126.8321   27723.3223  subfind_density()/subfind_density.c/97
  41 2985 0              Left    126.8267   27201.4885  subfind_density()/subfind_density.c/91
  41 2986 0             Right    126.8267   27328.3152  subfind_density()/subfind_density.c/92
  41 2987 0              Todo     15.8533   27344.1686  subfind_density()/subfind_density.c/93
  68 2983 0    DataIndexTable     21.8182   27015.2141  subfind_density()/subfind_density.c/87
  68 2984 0      DataNodeList     58.1818   27073.3958  subfind_density()/subfind_density.c/89
  68 2985 0              Left    126.8594   27200.2552  subfind_density()/subfind_density.c/91
  68 2986 0             Right    126.8594   27327.1146  subfind_density()/subfind_density.c/92
  68 2987 0              Todo     15.8574   27342.9721  subfind_density()/subfind_density.c/93
  67 2982 0         Dist2list    126.8589   26992.9337  subfind_density()/subfind_density.c/79
  67 2983 0    DataIndexTable     21.8182   27014.7519  subfind_density()/subfind_density.c/87
  67 2984 0      DataNodeList     58.1818   27072.9337  subfind_density()/subfind_density.c/89
  46 2987 0              Todo     15.8537   27342.9770  subfind_density()/subfind_density.c/93
  86 2984 0      DataNodeList     58.1818   27073.7383  subfind_density()/subfind_density.c/89
  86 2985 0              Left    126.8532   27200.5915  subfind_density()/subfind_density.c/91
  86 2986 0             Right    126.8532   27327.4447  subfind_density()/subfind_density.c/92
  59 2991 0    	  DensDataGet    121.3733   27844.6956  subfind_density()/subfind_density.c/161
  59 2992 0  	  DensDataResult     91.0300   27935.7256  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  41 2988 0             DM_Vx    126.8267   27470.9953  subfind_density()/subfind_density.c/95
  41 2989 0             DM_Vy    126.8267   27597.8221  subfind_density()/subfind_density.c/96
  41 2990 0             DM_Vz    126.8267   27724.6488  subfind_density()/subfind_density.c/97
  68 2988 0             DM_Vx    126.8594   27469.8315  subfind_density()/subfind_density.c/95
  68 2989 0             DM_Vy    126.8594   27596.6909  subfind_density()/subfind_density.c/96
  67 2985 0              Left    126.8589   27199.7926  subfind_density()/subfind_density.c/91
  67 2986 0             Right    126.8589   27326.6515  subfind_density()/subfind_density.c/92
  67 2987 0              Todo     15.8574   27342.5089  subfind_density()/subfind_density.c/93
  46 2988 0             DM_Vx    126.8294   27469.8064  subfind_density()/subfind_density.c/95
  86 2987 0              Todo     15.8567   27343.3013  subfind_density()/subfind_density.c/93
  86 2988 0             DM_Vx    126.8532   27470.1545  subfind_density()/subfind_density.c/95
  86 2989 0             DM_Vy    126.8532   27597.0077  subfind_density()/subfind_density.c/96

Task=59: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.2744 MB).
task 59: endrun called with an error level of 812


  41 2991 0    	  DensDataGet    131.7877   27856.4365  subfind_density()/subfind_density.c/161
  41 2992 0  	  DensDataResult     98.8407   27955.2772  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  68 2990 0             DM_Vz    126.8594   27723.5503  subfind_density()/subfind_density.c/97
  68 2991 0    	  DensDataGet    113.3677   27836.9179  subfind_density()/subfind_density.c/161
  67 2988 0             DM_Vx    126.8589   27469.3678  subfind_density()/subfind_density.c/95
  67 2989 0             DM_Vy    126.8589   27596.2267  subfind_density()/subfind_density.c/96
  46 2989 0             DM_Vy    126.8294   27596.6357  subfind_density()/subfind_density.c/96
  86 2990 0             DM_Vz    126.8532   27723.8609  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 59

Task=41: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=44.7228 MB).
task 41: endrun called with an error level of 812


  68 2992 0  	  DensDataResult     85.0258   27921.9437  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=68: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.0563 MB).
task 68: endrun called with an error level of 812


  67 2990 0             DM_Vz    126.8589   27723.0856  subfind_density()/subfind_density.c/97
  67 2991 0    	  DensDataGet    114.5656   27837.6512  subfind_density()/subfind_density.c/161
  67 2992 0  	  DensDataResult     85.9242   27923.5754  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  46 2990 0             DM_Vz    126.8294   27723.4651  subfind_density()/subfind_density.c/97
  86 2991 0    	  DensDataGet    127.5067   27851.3676  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 41
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 68

  46 2991 0    	  DensDataGet    119.0877   27842.5528  subfind_density()/subfind_density.c/161
  86 2992 0  	  DensDataResult     95.6300   27946.9976  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
Task=67: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.4246 MB).
task 67: endrun called with an error level of 812


  46 2992 0  	  DensDataResult     89.3158   27931.8686  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

application called MPI_Abort(MPI_COMM_WORLD, 812) - process 67

Task=46: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.1314 MB).
task 46: endrun called with an error level of 812



Task=86: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=53.0024 MB).
task 86: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 46
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 86
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  82 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  82 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  82 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  82 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  82 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  82 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  82 2973 0         FOF_PList    380.2625   18861.0643  fof_fof()/fof.c/249
  82 2974 0         FOF_GList    633.7708   19494.8351  fof_fof()/fof.c/337
  82 2975 0             Group     12.2520   19507.0871  fof_fof()/fof.c/380
  82 2976 0   DomainNodeIndex      0.3461   19507.4332  force_treeallocate()/forcetree.c/4800
  82 2977 0        Nodes_base   3737.5076   23244.9408  force_treeallocate()/forcetree.c/4803
  82 2978 0     Extnodes_base   3114.5897   26359.5305  force_treeallocate()/forcetree.c/4811
  82 2979 0          Nextnode    222.3012   26581.8317  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  82 2980 0            Father    221.9056   26803.7373  force_treeallocate()/forcetree.c/4827
  82 2981 0           Ngblist     63.3771   26867.1143  subfind_density()/subfind_density.c/78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  84 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  84 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  84 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  84 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  84 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  84 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  84 2973 0         FOF_PList    380.5791   18861.3809  fof_fof()/fof.c/249
  84 2974 0         FOF_GList    634.2985   19495.6794  fof_fof()/fof.c/337
  84 2975 0             Group     10.3220   19506.0014  fof_fof()/fof.c/380
  82 2982 0         Dist2list    126.7542   26993.8685  subfind_density()/subfind_density.c/79
  43 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  43 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  43 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  43 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  43 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  43 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  43 2973 0         FOF_PList    379.4174   18860.2191  fof_fof()/fof.c/249
  43 2974 0         FOF_GList    632.3623   19492.5814  fof_fof()/fof.c/337
  43 2975 0             Group     10.5845   19503.1659  fof_fof()/fof.c/380
  43 2976 0   DomainNodeIndex      0.3461   19503.5121  force_treeallocate()/forcetree.c/4800
  43 2977 0        Nodes_base   3737.5076   23241.0197  force_treeallocate()/forcetree.c/4803
  43 2978 0     Extnodes_base   3114.5897   26355.6094  force_treeallocate()/forcetree.c/4811
  43 2979 0          Nextnode    222.3012   26577.9105  force_treeallocate()/forcetree.c/4820
  43 2980 0            Father    221.9056   26799.8161  force_treeallocate()/forcetree.c/4827
  43 2981 0           Ngblist     63.2362   26863.0524  subfind_density()/subfind_density.c/78
  43 2982 0         Dist2list    126.4725   26989.5248  subfind_density()/subfind_density.c/79
  43 2983 0    DataIndexTable     21.8182   27011.3430  subfind_density()/subfind_density.c/87
  84 2976 0   DomainNodeIndex      0.3461   19506.3475  force_treeallocate()/forcetree.c/4800
  82 2983 0    DataIndexTable     21.8182   27015.6867  subfind_density()/subfind_density.c/87
  43 2984 0      DataNodeList     58.1818   27069.5248  subfind_density()/subfind_density.c/89
  43 2985 0              Left    126.4725   27195.9972  subfind_density()/subfind_density.c/91
  43 2986 0             Right    126.4725   27322.4697  subfind_density()/subfind_density.c/92
  43 2987 0              Todo     15.8091   27338.2787  subfind_density()/subfind_density.c/93
  43 2988 0             DM_Vx    126.4725   27464.7512  subfind_density()/subfind_density.c/95
  43 2989 0             DM_Vy    126.4725   27591.2236  subfind_density()/subfind_density.c/96
  84 2977 0        Nodes_base   3737.5076   23243.8551  force_treeallocate()/forcetree.c/4803
  84 2978 0     Extnodes_base   3114.5897   26358.4448  force_treeallocate()/forcetree.c/4811
  82 2984 0      DataNodeList     58.1818   27073.8685  subfind_density()/subfind_density.c/89
  82 2985 0              Left    126.7542   27200.6227  subfind_density()/subfind_density.c/91
  43 2990 0             DM_Vz    126.4725   27717.6961  subfind_density()/subfind_density.c/97
  43 2991 0    	  DensDataGet    126.8657   27844.5618  subfind_density()/subfind_density.c/161
  43 2992 0  	  DensDataResult     95.1493   27939.7111  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  84 2979 0          Nextnode    222.3012   26580.7460  force_treeallocate()/forcetree.c/4820
  84 2980 0            Father    221.9056   26802.6516  force_treeallocate()/forcetree.c/4827
  84 2981 0           Ngblist     63.4298   26866.0814  subfind_density()/subfind_density.c/78
  82 2986 0             Right    126.7542   27327.3768  subfind_density()/subfind_density.c/92

Task=43: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.2889 MB).
task 43: endrun called with an error level of 812


  84 2982 0         Dist2list    126.8597   26992.9411  subfind_density()/subfind_density.c/79
  84 2983 0    DataIndexTable     21.8182   27014.7593  subfind_density()/subfind_density.c/87
  82 2987 0              Todo     15.8443   27343.2211  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 43
  84 2984 0      DataNodeList     58.1818   27072.9411  subfind_density()/subfind_density.c/89
  84 2985 0              Left    126.8597   27199.8008  subfind_density()/subfind_density.c/91
  82 2988 0             DM_Vx    126.7542   27469.9753  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  84 2986 0             Right    126.8597   27326.6605  subfind_density()/subfind_density.c/92
  82 2989 0             DM_Vy    126.7542   27596.7294  subfind_density()/subfind_density.c/96
  66 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  66 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  66 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  61 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  61 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  61 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  61 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  61 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  61 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  61 2973 0         FOF_PList    380.5017   18861.3035  fof_fof()/fof.c/249
  61 2974 0         FOF_GList    634.1696   19495.4731  fof_fof()/fof.c/337
  61 2975 0             Group     11.5111   19506.9842  fof_fof()/fof.c/380
  61 2976 0   DomainNodeIndex      0.3461   19507.3304  force_treeallocate()/forcetree.c/4800
  61 2977 0        Nodes_base   3737.5076   23244.8380  force_treeallocate()/forcetree.c/4803
  61 2978 0     Extnodes_base   3114.5897   26359.4277  force_treeallocate()/forcetree.c/4811
  61 2979 0          Nextnode    222.3012   26581.7288  force_treeallocate()/forcetree.c/4820
  61 2980 0            Father    221.9056   26803.6344  force_treeallocate()/forcetree.c/4827
  61 2981 0           Ngblist     63.4170   26867.0514  subfind_density()/subfind_density.c/78
  61 2982 0         Dist2list    126.8339   26993.8853  subfind_density()/subfind_density.c/79
  61 2983 0    DataIndexTable     21.8182   27015.7035  subfind_density()/subfind_density.c/87
  61 2984 0      DataNodeList     58.1818   27073.8853  subfind_density()/subfind_density.c/89
  61 2985 0              Left    126.8339   27200.7192  subfind_density()/subfind_density.c/91
  84 2987 0              Todo     15.8575   27342.5179  subfind_density()/subfind_density.c/93
  82 2990 0             DM_Vz    126.7542   27723.4836  subfind_density()/subfind_density.c/97
  66 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  66 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  42 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  42 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  42 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  42 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  42 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  42 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  61 2986 0             Right    126.8339   27327.5531  subfind_density()/subfind_density.c/92
  61 2987 0              Todo     15.8542   27343.4073  subfind_density()/subfind_density.c/93
  61 2988 0             DM_Vx    126.8339   27470.2413  subfind_density()/subfind_density.c/95
  61 2989 0             DM_Vy    126.8339   27597.0752  subfind_density()/subfind_density.c/96
  61 2990 0             DM_Vz    126.8339   27723.9091  subfind_density()/subfind_density.c/97
  61 2991 0    	  DensDataGet    114.7086   27838.6176  subfind_density()/subfind_density.c/161
  61 2992 0  	  DensDataResult     86.0314   27924.6491  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  84 2988 0             DM_Vx    126.8597   27469.3776  subfind_density()/subfind_density.c/95
  82 2991 0    	  DensDataGet    111.3830   27834.8666  subfind_density()/subfind_density.c/161
  66 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  66 2973 0         FOF_PList    380.5675   18861.3693  fof_fof()/fof.c/249
  66 2974 0         FOF_GList    634.2792   19495.6485  fof_fof()/fof.c/337
  42 2973 0         FOF_PList    380.5671   18861.3689  fof_fof()/fof.c/249
  42 2974 0         FOF_GList    634.2785   19495.6473  fof_fof()/fof.c/337

Task=61: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.3509 MB).
task 61: endrun called with an error level of 812


  84 2989 0             DM_Vy    126.8597   27596.2373  subfind_density()/subfind_density.c/96
  84 2990 0             DM_Vz    126.8597   27723.0970  subfind_density()/subfind_density.c/97
  82 2992 0  	  DensDataResult     83.5372   27918.4038  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  66 2975 0             Group     11.7186   19507.3672  fof_fof()/fof.c/380
  66 2976 0   DomainNodeIndex      0.3461   19507.7133  force_treeallocate()/forcetree.c/4800
  42 2975 0             Group     11.5194   19507.1667  fof_fof()/fof.c/380
  42 2976 0   DomainNodeIndex      0.3461   19507.5128  force_treeallocate()/forcetree.c/4800
  42 2977 0        Nodes_base   3737.5076   23245.0205  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 61
  84 2991 0    	  DensDataGet    129.2147   27852.3117  subfind_density()/subfind_density.c/161

Task=82: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.5962 MB).
task 82: endrun called with an error level of 812


  66 2977 0        Nodes_base   3737.5076   23245.2209  force_treeallocate()/forcetree.c/4803
  66 2978 0     Extnodes_base   3114.5897   26359.8106  force_treeallocate()/forcetree.c/4811
  42 2978 0     Extnodes_base   3114.5897   26359.6102  force_treeallocate()/forcetree.c/4811
  42 2979 0          Nextnode    222.3012   26581.9113  force_treeallocate()/forcetree.c/4820
  84 2992 0  	  DensDataResult     96.9110   27949.2227  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 82
  66 2979 0          Nextnode    222.3012   26582.1118  force_treeallocate()/forcetree.c/4820
  66 2980 0            Father    221.9056   26804.0174  force_treeallocate()/forcetree.c/4827
  42 2980 0            Father    221.9056   26803.8169  force_treeallocate()/forcetree.c/4827
  42 2981 0           Ngblist     63.4278   26867.2448  subfind_density()/subfind_density.c/78
  42 2982 0         Dist2list    126.8557   26994.1005  subfind_density()/subfind_density.c/79

  66 2981 0           Ngblist     63.4279   26867.4453  subfind_density()/subfind_density.c/78
  42 2983 0    DataIndexTable     21.8182   27015.9186  subfind_density()/subfind_density.c/87
  42 2984 0      DataNodeList     58.1818   27074.1004  subfind_density()/subfind_density.c/89
  42 2985 0              Left    126.8557   27200.9561  subfind_density()/subfind_density.c/91

Task=84: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.7773 MB).
task 84: endrun called with an error level of 812


  66 2982 0         Dist2list    126.8558   26994.3011  subfind_density()/subfind_density.c/79
  42 2986 0             Right    126.8557   27327.8118  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 84
  66 2983 0    DataIndexTable     21.8182   27016.1193  subfind_density()/subfind_density.c/87
  42 2987 0              Todo     15.8570   27343.6688  subfind_density()/subfind_density.c/93
  66 2984 0      DataNodeList     58.1818   27074.3011  subfind_density()/subfind_density.c/89
  42 2988 0             DM_Vx    126.8557   27470.5245  subfind_density()/subfind_density.c/95
  66 2985 0              Left    126.8558   27201.1569  subfind_density()/subfind_density.c/91
  42 2989 0             DM_Vy    126.8557   27597.3801  subfind_density()/subfind_density.c/96
  66 2986 0             Right    126.8558   27328.0128  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  42 2990 0             DM_Vz    126.8557   27724.2358  subfind_density()/subfind_density.c/97
  66 2987 0              Todo     15.8570   27343.8698  subfind_density()/subfind_density.c/93
  93 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  93 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  93 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  93 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  93 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  93 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  93 2973 0         FOF_PList    380.4511   18861.2529  fof_fof()/fof.c/249
  93 2974 0         FOF_GList    634.0852   19495.3380  fof_fof()/fof.c/337
  93 2975 0             Group     12.6556   19507.9936  fof_fof()/fof.c/380
  93 2976 0   DomainNodeIndex      0.3461   19508.3398  force_treeallocate()/forcetree.c/4800
  93 2977 0        Nodes_base   3737.5076   23245.8474  force_treeallocate()/forcetree.c/4803
  93 2978 0     Extnodes_base   3114.5897   26360.4371  force_treeallocate()/forcetree.c/4811
  93 2979 0          Nextnode    222.3012   26582.7382  force_treeallocate()/forcetree.c/4820
  93 2980 0            Father    221.9056   26804.6438  force_treeallocate()/forcetree.c/4827
  93 2981 0           Ngblist     63.4085   26868.0523  subfind_density()/subfind_density.c/78
  93 2982 0         Dist2list    126.8170   26994.8694  subfind_density()/subfind_density.c/79
  42 2991 0    	  DensDataGet    115.0695   27839.3054  subfind_density()/subfind_density.c/161
  66 2988 0             DM_Vx    126.8558   27470.7256  subfind_density()/subfind_density.c/95
  93 2983 0    DataIndexTable     21.8182   27016.6875  subfind_density()/subfind_density.c/87
  93 2984 0      DataNodeList     58.1818   27074.8693  subfind_density()/subfind_density.c/89
  93 2985 0              Left    126.8170   27201.6864  subfind_density()/subfind_density.c/91
  93 2986 0             Right    126.8170   27328.5034  subfind_density()/subfind_density.c/92
  93 2987 0              Todo     15.8521   27344.3555  subfind_density()/subfind_density.c/93
  42 2992 0  	  DensDataResult     86.3021   27925.6075  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  66 2989 0             DM_Vy    126.8558   27597.5815  subfind_density()/subfind_density.c/96
  93 2988 0             DM_Vx    126.8170   27471.1726  subfind_density()/subfind_density.c/95
  93 2989 0             DM_Vy    126.8170   27597.9896  subfind_density()/subfind_density.c/96
  93 2990 0             DM_Vz    126.8170   27724.8066  subfind_density()/subfind_density.c/97

Task=42: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.3925 MB).
  66 2990 0             DM_Vz    126.8558   27724.4373  subfind_density()/subfind_density.c/97
  93 2991 0    	  DensDataGet    140.6907   27865.4974  subfind_density()/subfind_density.c/161
  93 2992 0  	  DensDataResult    105.5181   27971.0154  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 42: endrun called with an error level of 812


  66 2991 0    	  DensDataGet    128.9719   27853.4092  subfind_density()/subfind_density.c/161

Task=93: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=28.9846 MB).
task 93: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 42
  66 2992 0  	  DensDataResult     96.7289   27950.1382  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=66: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.8618 MB).
task 66: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  35 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  35 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  35 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  35 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  35 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  35 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  35 2973 0         FOF_PList    380.2746   18861.0763  fof_fof()/fof.c/249
  35 2974 0         FOF_GList    633.7909   19494.8673  fof_fof()/fof.c/337
  35 2975 0             Group     10.6375   19505.5047  fof_fof()/fof.c/380
  35 2976 0   DomainNodeIndex      0.3461   19505.8509  force_treeallocate()/forcetree.c/4800
  35 2977 0        Nodes_base   3737.5076   23243.3585  force_treeallocate()/forcetree.c/4803
  35 2978 0     Extnodes_base   3114.5897   26357.9482  force_treeallocate()/forcetree.c/4811
  35 2979 0          Nextnode    222.3012   26580.2493  force_treeallocate()/forcetree.c/4820
  35 2980 0            Father    221.9056   26802.1549  force_treeallocate()/forcetree.c/4827
  35 2981 0           Ngblist     63.3791   26865.5340  subfind_density()/subfind_density.c/78
  35 2982 0         Dist2list    126.7582   26992.2922  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 66
  92 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  92 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  92 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  92 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  92 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  92 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  35 2983 0    DataIndexTable     21.8182   27014.1104  subfind_density()/subfind_density.c/87
  35 2984 0      DataNodeList     58.1818   27072.2922  subfind_density()/subfind_density.c/89
  35 2985 0              Left    126.7582   27199.0504  subfind_density()/subfind_density.c/91
  35 2986 0             Right    126.7582   27325.8086  subfind_density()/subfind_density.c/92
  35 2987 0              Todo     15.8448   27341.6533  subfind_density()/subfind_density.c/93
  92 2973 0         FOF_PList    380.0814   18860.8832  fof_fof()/fof.c/249
  92 2974 0         FOF_GList    633.4690   19494.3521  fof_fof()/fof.c/337
  92 2975 0             Group     10.3220   19504.6742  fof_fof()/fof.c/380
  35 2988 0             DM_Vx    126.7582   27468.4115  subfind_density()/subfind_density.c/95
  35 2989 0             DM_Vy    126.7582   27595.1697  subfind_density()/subfind_density.c/96
  35 2990 0             DM_Vz    126.7582   27721.9279  subfind_density()/subfind_density.c/97
  35 2991 0    	  DensDataGet    118.3106   27840.2385  subfind_density()/subfind_density.c/161
  92 2976 0   DomainNodeIndex      0.3461   19505.0203  force_treeallocate()/forcetree.c/4800
  92 2977 0        Nodes_base   3737.5076   23242.5279  force_treeallocate()/forcetree.c/4803
  92 2978 0     Extnodes_base   3114.5897   26357.1176  force_treeallocate()/forcetree.c/4811
  35 2992 0  	  DensDataResult     88.7330   27928.9715  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=35: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.0285 MB).
  92 2979 0          Nextnode    222.3012   26579.4188  force_treeallocate()/forcetree.c/4820
  92 2980 0            Father    221.9056   26801.3244  force_treeallocate()/forcetree.c/4827
task 35: endrun called with an error level of 812


  92 2981 0           Ngblist     63.3469   26864.6713  subfind_density()/subfind_density.c/78
  92 2982 0         Dist2list    126.6938   26991.3651  subfind_density()/subfind_density.c/79
  92 2983 0    DataIndexTable     21.8182   27013.1832  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 35
  92 2984 0      DataNodeList     58.1818   27071.3650  subfind_density()/subfind_density.c/89
  92 2985 0              Left    126.6938   27198.0588  subfind_density()/subfind_density.c/91
  92 2986 0             Right    126.6938   27324.7526  subfind_density()/subfind_density.c/92
  92 2987 0              Todo     15.8367   27340.5893  subfind_density()/subfind_density.c/93
  92 2988 0             DM_Vx    126.6938   27467.2831  subfind_density()/subfind_density.c/95
  92 2989 0             DM_Vy    126.6938   27593.9769  subfind_density()/subfind_density.c/96
  92 2990 0             DM_Vz    126.6938   27720.6707  subfind_density()/subfind_density.c/97
  92 2991 0    	  DensDataGet    115.9762   27836.6469  subfind_density()/subfind_density.c/161
  92 2992 0  	  DensDataResult     86.9821   27923.6291  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=92: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.3709 MB).
task 92: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  87 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  87 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  87 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  87 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  87 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  87 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  87 2973 0         FOF_PList    380.5205   18861.3223  fof_fof()/fof.c/249
  87 2974 0         FOF_GList    634.2008   19495.5231  fof_fof()/fof.c/337
  87 2975 0             Group     11.5402   19507.0633  fof_fof()/fof.c/380
  87 2976 0   DomainNodeIndex      0.3461   19507.4094  force_treeallocate()/forcetree.c/4800
  87 2977 0        Nodes_base   3737.5076   23244.9170  force_treeallocate()/forcetree.c/4803
  87 2978 0     Extnodes_base   3114.5897   26359.5067  force_treeallocate()/forcetree.c/4811
  87 2979 0          Nextnode    222.3012   26581.8079  force_treeallocate()/forcetree.c/4820
  87 2980 0            Father    221.9056   26803.7135  force_treeallocate()/forcetree.c/4827
  87 2981 0           Ngblist     63.4201   26867.1335  subfind_density()/subfind_density.c/78
  87 2982 0         Dist2list    126.8402   26993.9737  subfind_density()/subfind_density.c/79
  87 2983 0    DataIndexTable     21.8182   27015.7919  subfind_density()/subfind_density.c/87
  87 2984 0      DataNodeList     58.1818   27073.9737  subfind_density()/subfind_density.c/89
  87 2985 0              Left    126.8402   27200.8138  subfind_density()/subfind_density.c/91
  87 2986 0             Right    126.8402   27327.6540  subfind_density()/subfind_density.c/92
  87 2987 0              Todo     15.8550   27343.5090  subfind_density()/subfind_density.c/93
  87 2988 0             DM_Vx    126.8402   27470.3492  subfind_density()/subfind_density.c/95
  87 2989 0             DM_Vy    126.8402   27597.1894  subfind_density()/subfind_density.c/96
  87 2990 0             DM_Vz    126.8402   27724.0295  subfind_density()/subfind_density.c/97
  87 2991 0    	  DensDataGet    134.0647   27858.0942  subfind_density()/subfind_density.c/161
  87 2992 0  	  DensDataResult    100.5485   27958.6427  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=87: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=41.3573 MB).
task 87: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  89 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  89 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  89 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  89 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  89 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  89 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  89 2973 0         FOF_PList    380.2175   18861.0192  fof_fof()/fof.c/249
  89 2974 0         FOF_GList    633.6958   19494.7150  fof_fof()/fof.c/337
  89 2975 0             Group     12.6089   19507.3239  fof_fof()/fof.c/380
  89 2976 0   DomainNodeIndex      0.3461   19507.6700  force_treeallocate()/forcetree.c/4800
  89 2977 0        Nodes_base   3737.5076   23245.1776  force_treeallocate()/forcetree.c/4803
  89 2978 0     Extnodes_base   3114.5897   26359.7673  force_treeallocate()/forcetree.c/4811
  89 2979 0          Nextnode    222.3012   26582.0685  force_treeallocate()/forcetree.c/4820
  89 2980 0            Father    221.9056   26803.9741  force_treeallocate()/forcetree.c/4827
  89 2981 0           Ngblist     63.3696   26867.3437  subfind_density()/subfind_density.c/78
  89 2982 0         Dist2list    126.7392   26994.0828  subfind_density()/subfind_density.c/79
  38 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  38 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  38 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  38 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  38 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  38 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  38 2973 0         FOF_PList    380.4683   18861.2701  fof_fof()/fof.c/249
  38 2974 0         FOF_GList    634.1138   19495.3839  fof_fof()/fof.c/337
  38 2975 0             Group     11.0732   19506.4571  fof_fof()/fof.c/380
  38 2976 0   DomainNodeIndex      0.3461   19506.8033  force_treeallocate()/forcetree.c/4800
  38 2977 0        Nodes_base   3737.5076   23244.3109  force_treeallocate()/forcetree.c/4803
  89 2983 0    DataIndexTable     21.8182   27015.9010  subfind_density()/subfind_density.c/87
  89 2984 0      DataNodeList     58.1818   27074.0828  subfind_density()/subfind_density.c/89
  89 2985 0              Left    126.7392   27200.8219  subfind_density()/subfind_density.c/91
  89 2986 0             Right    126.7392   27327.5611  subfind_density()/subfind_density.c/92
  89 2987 0              Todo     15.8424   27343.4035  subfind_density()/subfind_density.c/93
  38 2978 0     Extnodes_base   3114.5897   26358.9006  force_treeallocate()/forcetree.c/4811
  38 2979 0          Nextnode    222.3012   26581.2017  force_treeallocate()/forcetree.c/4820
  89 2988 0             DM_Vx    126.7392   27470.1426  subfind_density()/subfind_density.c/95
  89 2989 0             DM_Vy    126.7392   27596.8818  subfind_density()/subfind_density.c/96
  89 2990 0             DM_Vz    126.7392   27723.6209  subfind_density()/subfind_density.c/97
  38 2980 0            Father    221.9056   26803.1073  force_treeallocate()/forcetree.c/4827
  38 2981 0           Ngblist     63.4114   26866.5187  subfind_density()/subfind_density.c/78
  38 2982 0         Dist2list    126.8228   26993.3415  subfind_density()/subfind_density.c/79
  89 2991 0    	  DensDataGet    116.6813   27840.3023  subfind_density()/subfind_density.c/161
  89 2992 0  	  DensDataResult     87.5110   27927.8133  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  38 2983 0    DataIndexTable     21.8182   27015.1596  subfind_density()/subfind_density.c/87
  38 2984 0      DataNodeList     58.1818   27073.3414  subfind_density()/subfind_density.c/89
  38 2985 0              Left    126.8228   27200.1642  subfind_density()/subfind_density.c/91
  38 2986 0             Right    126.8228   27326.9870  subfind_density()/subfind_density.c/92

Task=89: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.1867 MB).
task 89: endrun called with an error level of 812


  38 2987 0              Todo     15.8529   27342.8398  subfind_density()/subfind_density.c/93
  38 2988 0             DM_Vx    126.8228   27469.6626  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 89
  38 2989 0             DM_Vy    126.8228   27596.4853  subfind_density()/subfind_density.c/96
  38 2990 0             DM_Vz    126.8228   27723.3081  subfind_density()/subfind_density.c/97
  38 2991 0    	  DensDataGet    144.9734   27868.2815  subfind_density()/subfind_density.c/161
  38 2992 0  	  DensDataResult    108.7301   27977.0116  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=38: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=22.9884 MB).
task 38: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 38
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  53 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  53 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  53 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  53 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  53 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  53 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  53 2973 0         FOF_PList    380.5868   18861.3886  fof_fof()/fof.c/249
  53 2974 0         FOF_GList    634.3114   19495.7000  fof_fof()/fof.c/337
  53 2975 0             Group     10.7256   19506.4256  fof_fof()/fof.c/380
  53 2976 0   DomainNodeIndex      0.3461   19506.7718  force_treeallocate()/forcetree.c/4800
  53 2977 0        Nodes_base   3737.5076   23244.2794  force_treeallocate()/forcetree.c/4803
  53 2978 0     Extnodes_base   3114.5897   26358.8691  force_treeallocate()/forcetree.c/4811
  53 2979 0          Nextnode    222.3012   26581.1702  force_treeallocate()/forcetree.c/4820
  53 2980 0            Father    221.9056   26803.0758  force_treeallocate()/forcetree.c/4827
  53 2981 0           Ngblist     63.4311   26866.5070  subfind_density()/subfind_density.c/78
  53 2982 0         Dist2list    126.8623   26993.3692  subfind_density()/subfind_density.c/79
  74 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  74 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  74 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  74 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  74 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  74 2973 0         FOF_PList    380.5827   18861.3845  fof_fof()/fof.c/249
  74 2974 0         FOF_GList    634.3046   19495.6891  fof_fof()/fof.c/337
  74 2975 0             Group     11.7010   19507.3901  fof_fof()/fof.c/380
  74 2976 0   DomainNodeIndex      0.3461   19507.7362  force_treeallocate()/forcetree.c/4800
  74 2977 0        Nodes_base   3737.5076   23245.2439  force_treeallocate()/forcetree.c/4803
  74 2978 0     Extnodes_base   3114.5897   26359.8336  force_treeallocate()/forcetree.c/4811
  53 2983 0    DataIndexTable     21.8182   27015.1874  subfind_density()/subfind_density.c/87
  53 2984 0      DataNodeList     58.1818   27073.3692  subfind_density()/subfind_density.c/89
  53 2985 0              Left    126.8623   27200.2315  subfind_density()/subfind_density.c/91
  53 2986 0             Right    126.8623   27327.0938  subfind_density()/subfind_density.c/92
  74 2979 0          Nextnode    222.3012   26582.1347  force_treeallocate()/forcetree.c/4820
  74 2980 0            Father    221.9056   26804.0403  force_treeallocate()/forcetree.c/4827
  74 2981 0           Ngblist     63.4305   26867.4708  subfind_density()/subfind_density.c/78
  74 2982 0         Dist2list    126.8609   26994.3317  subfind_density()/subfind_density.c/79
  53 2987 0              Todo     15.8578   27342.9515  subfind_density()/subfind_density.c/93
  53 2988 0             DM_Vx    126.8623   27469.8138  subfind_density()/subfind_density.c/95
  53 2989 0             DM_Vy    126.8623   27596.6761  subfind_density()/subfind_density.c/96
  53 2990 0             DM_Vz    126.8623   27723.5384  subfind_density()/subfind_density.c/97
  74 2983 0    DataIndexTable     21.8182   27016.1499  subfind_density()/subfind_density.c/87
  74 2984 0      DataNodeList     58.1818   27074.3316  subfind_density()/subfind_density.c/89
  74 2985 0              Left    126.8609   27201.1926  subfind_density()/subfind_density.c/91
  53 2991 0    	  DensDataGet    117.7802   27841.3186  subfind_density()/subfind_density.c/161
  53 2992 0  	  DensDataResult     88.3352   27929.6537  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  74 2986 0             Right    126.8609   27328.0535  subfind_density()/subfind_density.c/92
  74 2987 0              Todo     15.8576   27343.9111  subfind_density()/subfind_density.c/93
  74 2988 0             DM_Vx    126.8609   27470.7720  subfind_density()/subfind_density.c/95

Task=53: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.3463 MB).
task 53: endrun called with an error level of 812


  74 2989 0             DM_Vy    126.8609   27597.6329  subfind_density()/subfind_density.c/96
  74 2990 0             DM_Vz    126.8609   27724.4939  subfind_density()/subfind_density.c/97
  74 2991 0    	  DensDataGet    108.8290   27833.3228  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 53
  74 2992 0  	  DensDataResult     81.6217   27914.9446  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=74: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.0554 MB).
task 74: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 74
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  75 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  75 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  75 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  75 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  75 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  75 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  75 2973 0         FOF_PList    380.4211   18861.2229  fof_fof()/fof.c/249
  75 2974 0         FOF_GList    634.0351   19495.2580  fof_fof()/fof.c/337
  75 2975 0             Group     11.2600   19506.5180  fof_fof()/fof.c/380
  75 2976 0   DomainNodeIndex      0.3461   19506.8641  force_treeallocate()/forcetree.c/4800
  75 2977 0        Nodes_base   3737.5076   23244.3717  force_treeallocate()/forcetree.c/4803
  75 2978 0     Extnodes_base   3114.5897   26358.9614  force_treeallocate()/forcetree.c/4811
  75 2979 0          Nextnode    222.3012   26581.2626  force_treeallocate()/forcetree.c/4820
  75 2980 0            Father    221.9056   26803.1682  force_treeallocate()/forcetree.c/4827
  75 2981 0           Ngblist     63.4035   26866.5717  subfind_density()/subfind_density.c/78
  75 2982 0         Dist2list    126.8070   26993.3787  subfind_density()/subfind_density.c/79
  75 2983 0    DataIndexTable     21.8182   27015.1969  subfind_density()/subfind_density.c/87
  75 2984 0      DataNodeList     58.1818   27073.3787  subfind_density()/subfind_density.c/89
  75 2985 0              Left    126.8070   27200.1857  subfind_density()/subfind_density.c/91
  75 2986 0             Right    126.8070   27326.9927  subfind_density()/subfind_density.c/92
  75 2987 0              Todo     15.8509   27342.8436  subfind_density()/subfind_density.c/93
  75 2988 0             DM_Vx    126.8070   27469.6506  subfind_density()/subfind_density.c/95
  75 2989 0             DM_Vy    126.8070   27596.4576  subfind_density()/subfind_density.c/96
  75 2990 0             DM_Vz    126.8070   27723.2647  subfind_density()/subfind_density.c/97
  75 2991 0    	  DensDataGet    116.6280   27839.8927  subfind_density()/subfind_density.c/161
  75 2992 0  	  DensDataResult     87.4710   27927.3637  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=75: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.6363 MB).
task 75: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 75
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  32 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  32 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  32 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  32 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  32 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  32 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  32 2973 0         FOF_PList    380.4607   18861.2625  fof_fof()/fof.c/249
  32 2974 0         FOF_GList    634.1011   19495.3636  fof_fof()/fof.c/337
  32 2975 0             Group     12.4948   19507.8584  fof_fof()/fof.c/380
  32 2976 0   DomainNodeIndex      0.3461   19508.2045  force_treeallocate()/forcetree.c/4800
  95 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  95 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  95 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  95 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  95 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  95 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  95 2973 0         FOF_PList    380.1014   18860.9032  fof_fof()/fof.c/249
  95 2974 0         FOF_GList    633.5023   19494.4054  fof_fof()/fof.c/337
  95 2975 0             Group     10.4735   19504.8789  fof_fof()/fof.c/380
  95 2976 0   DomainNodeIndex      0.3461   19505.2251  force_treeallocate()/forcetree.c/4800
  95 2977 0        Nodes_base   3737.5076   23242.7327  force_treeallocate()/forcetree.c/4803
  95 2978 0     Extnodes_base   3114.5897   26357.3224  force_treeallocate()/forcetree.c/4811
  95 2979 0          Nextnode    222.3012   26579.6236  force_treeallocate()/forcetree.c/4820
  95 2980 0            Father    221.9056   26801.5291  force_treeallocate()/forcetree.c/4827
  95 2981 0           Ngblist     63.3502   26864.8794  subfind_density()/subfind_density.c/78
  95 2982 0         Dist2list    126.7005   26991.5798  subfind_density()/subfind_density.c/79
  95 2983 0    DataIndexTable     21.8182   27013.3980  subfind_density()/subfind_density.c/87
  32 2977 0        Nodes_base   3737.5076   23245.7121  force_treeallocate()/forcetree.c/4803
  32 2978 0     Extnodes_base   3114.5897   26360.3018  force_treeallocate()/forcetree.c/4811
  95 2984 0      DataNodeList     58.1818   27071.5798  subfind_density()/subfind_density.c/89
  95 2985 0              Left    126.7005   27198.2802  subfind_density()/subfind_density.c/91
  95 2986 0             Right    126.7005   27324.9807  subfind_density()/subfind_density.c/92
  95 2987 0              Todo     15.8376   27340.8183  subfind_density()/subfind_density.c/93
  95 2988 0             DM_Vx    126.7005   27467.5187  subfind_density()/subfind_density.c/95
  95 2989 0             DM_Vy    126.7005   27594.2192  subfind_density()/subfind_density.c/96
  32 2979 0          Nextnode    222.3012   26582.6030  force_treeallocate()/forcetree.c/4820
  32 2980 0            Father    221.9056   26804.5086  force_treeallocate()/forcetree.c/4827
  32 2981 0           Ngblist     63.4101   26867.9187  subfind_density()/subfind_density.c/78
  95 2990 0             DM_Vz    126.7005   27720.9196  subfind_density()/subfind_density.c/97
  95 2991 0    	  DensDataGet    113.0222   27833.9418  subfind_density()/subfind_density.c/161
  95 2992 0  	  DensDataResult     84.7667   27918.7085  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  32 2982 0         Dist2list    126.8202   26994.7389  subfind_density()/subfind_density.c/79
  32 2983 0    DataIndexTable     21.8182   27016.5571  subfind_density()/subfind_density.c/87
  32 2984 0      DataNodeList     58.1818   27074.7389  subfind_density()/subfind_density.c/89

Task=95: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.2915 MB).
task 95: endrun called with an error level of 812


  32 2985 0              Left    126.8202   27201.5591  subfind_density()/subfind_density.c/91
  32 2986 0             Right    126.8202   27328.3793  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 95
  32 2987 0              Todo     15.8525   27344.2319  subfind_density()/subfind_density.c/93
  32 2988 0             DM_Vx    126.8202   27471.0521  subfind_density()/subfind_density.c/95
  32 2989 0             DM_Vy    126.8202   27597.8723  subfind_density()/subfind_density.c/96
  32 2990 0             DM_Vz    126.8202   27724.6926  subfind_density()/subfind_density.c/97
  32 2991 0    	  DensDataGet    111.1749   27835.8674  subfind_density()/subfind_density.c/161
  32 2992 0  	  DensDataResult     83.3811   27919.2486  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=32: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.7514 MB).
task 32: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 32
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 32:
task 32: endrun called with an error level of 124


  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 32
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  88 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  88 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  88 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  88 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  88 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  88 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  88 2973 0         FOF_PList    380.5105   18861.3123  fof_fof()/fof.c/249
  88 2974 0         FOF_GList    634.1842   19495.4966  fof_fof()/fof.c/337
  88 2975 0             Group     11.8961   19507.3926  fof_fof()/fof.c/380
  88 2976 0   DomainNodeIndex      0.3461   19507.7387  force_treeallocate()/forcetree.c/4800
  88 2977 0        Nodes_base   3737.5076   23245.2464  force_treeallocate()/forcetree.c/4803
  88 2978 0     Extnodes_base   3114.5897   26359.8361  force_treeallocate()/forcetree.c/4811
  88 2979 0          Nextnode    222.3012   26582.1372  force_treeallocate()/forcetree.c/4820
  88 2980 0            Father    221.9056   26804.0428  force_treeallocate()/forcetree.c/4827
  88 2981 0           Ngblist     63.4184   26867.4612  subfind_density()/subfind_density.c/78
  88 2982 0         Dist2list    126.8368   26994.2981  subfind_density()/subfind_density.c/79
  88 2983 0    DataIndexTable     21.8182   27016.1163  subfind_density()/subfind_density.c/87
  88 2984 0      DataNodeList     58.1818   27074.2980  subfind_density()/subfind_density.c/89
  88 2985 0              Left    126.8368   27201.1349  subfind_density()/subfind_density.c/91
  88 2986 0             Right    126.8368   27327.9717  subfind_density()/subfind_density.c/92
  88 2987 0              Todo     15.8546   27343.8263  subfind_density()/subfind_density.c/93
  88 2988 0             DM_Vx    126.8368   27470.6632  subfind_density()/subfind_density.c/95
  88 2989 0             DM_Vy    126.8368   27597.5000  subfind_density()/subfind_density.c/96
  88 2990 0             DM_Vz    126.8368   27724.3369  subfind_density()/subfind_density.c/97
  88 2991 0    	  DensDataGet    122.2901   27846.6270  subfind_density()/subfind_density.c/161
  88 2992 0  	  DensDataResult     91.7176   27938.3446  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=88: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.6554 MB).
task 88: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 88
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 88:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
task 88: endrun called with an error level of 124


    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 88
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  33 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  33 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  33 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  33 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  33 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  33 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  33 2973 0         FOF_PList    380.5781   18861.3799  fof_fof()/fof.c/249
  33 2974 0         FOF_GList    634.2968   19495.6767  fof_fof()/fof.c/337
  33 2975 0             Group     12.3121   19507.9889  fof_fof()/fof.c/380
  33 2976 0   DomainNodeIndex      0.3461   19508.3350  force_treeallocate()/forcetree.c/4800
  33 2977 0        Nodes_base   3737.5076   23245.8426  force_treeallocate()/forcetree.c/4803
  33 2978 0     Extnodes_base   3114.5897   26360.4323  force_treeallocate()/forcetree.c/4811
  33 2979 0          Nextnode    222.3012   26582.7335  force_treeallocate()/forcetree.c/4820
  33 2980 0            Father    221.9056   26804.6391  force_treeallocate()/forcetree.c/4827
  33 2981 0           Ngblist     63.4297   26868.0687  subfind_density()/subfind_density.c/78
  33 2982 0         Dist2list    126.8594   26994.9281  subfind_density()/subfind_density.c/79
  33 2983 0    DataIndexTable     21.8182   27016.7463  subfind_density()/subfind_density.c/87
  33 2984 0      DataNodeList     58.1818   27074.9281  subfind_density()/subfind_density.c/89
  33 2985 0              Left    126.8594   27201.7875  subfind_density()/subfind_density.c/91
  33 2986 0             Right    126.8594   27328.6468  subfind_density()/subfind_density.c/92
  33 2987 0              Todo     15.8574   27344.5042  subfind_density()/subfind_density.c/93
  33 2988 0             DM_Vx    126.8594   27471.3636  subfind_density()/subfind_density.c/95
  33 2989 0             DM_Vy    126.8594   27598.2230  subfind_density()/subfind_density.c/96
  33 2990 0             DM_Vz    126.8594   27725.0823  subfind_density()/subfind_density.c/97
  33 2991 0    	  DensDataGet    128.5205   27853.6029  subfind_density()/subfind_density.c/161
  33 2992 0  	  DensDataResult     96.3904   27949.9932  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=33: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.0068 MB).
task 33: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 33
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  50 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  50 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  50 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  50 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  50 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  50 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  50 2973 0         FOF_PList    380.3779   18861.1797  fof_fof()/fof.c/249
  50 2974 0         FOF_GList    633.9632   19495.1429  fof_fof()/fof.c/337
  77 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  77 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  77 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  77 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  77 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  77 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  77 2973 0         FOF_PList    380.5155   18861.3172  fof_fof()/fof.c/249
  77 2974 0         FOF_GList    634.1924   19495.5097  fof_fof()/fof.c/337
  77 2975 0             Group     11.7726   19507.2823  fof_fof()/fof.c/380
  77 2976 0   DomainNodeIndex      0.3461   19507.6284  force_treeallocate()/forcetree.c/4800
  77 2977 0        Nodes_base   3737.5076   23245.1360  force_treeallocate()/forcetree.c/4803
  77 2978 0     Extnodes_base   3114.5897   26359.7257  force_treeallocate()/forcetree.c/4811
  77 2979 0          Nextnode    222.3012   26582.0269  force_treeallocate()/forcetree.c/4820
  77 2980 0            Father    221.9056   26803.9325  force_treeallocate()/forcetree.c/4827
  50 2975 0             Group     11.1282   19506.2711  fof_fof()/fof.c/380
  50 2976 0   DomainNodeIndex      0.3461   19506.6172  force_treeallocate()/forcetree.c/4800
  50 2977 0        Nodes_base   3737.5076   23244.1248  force_treeallocate()/forcetree.c/4803
  77 2981 0           Ngblist     63.4192   26867.3517  subfind_density()/subfind_density.c/78
  77 2982 0         Dist2list    126.8385   26994.1902  subfind_density()/subfind_density.c/79
  77 2983 0    DataIndexTable     21.8182   27016.0084  subfind_density()/subfind_density.c/87
  77 2984 0      DataNodeList     58.1818   27074.1902  subfind_density()/subfind_density.c/89
  50 2978 0     Extnodes_base   3114.5897   26358.7145  force_treeallocate()/forcetree.c/4811
  50 2979 0          Nextnode    222.3012   26581.0157  force_treeallocate()/forcetree.c/4820
  50 2980 0            Father    221.9056   26802.9213  force_treeallocate()/forcetree.c/4827
  77 2985 0              Left    126.8385   27201.0286  subfind_density()/subfind_density.c/91
  77 2986 0             Right    126.8385   27327.8671  subfind_density()/subfind_density.c/92
  77 2987 0              Todo     15.8548   27343.7219  subfind_density()/subfind_density.c/93
  50 2981 0           Ngblist     63.3963   26866.3176  subfind_density()/subfind_density.c/78
  50 2982 0         Dist2list    126.7926   26993.1102  subfind_density()/subfind_density.c/79
  50 2983 0    DataIndexTable     21.8182   27014.9284  subfind_density()/subfind_density.c/87
  77 2988 0             DM_Vx    126.8385   27470.5604  subfind_density()/subfind_density.c/95
  77 2989 0             DM_Vy    126.8385   27597.3989  subfind_density()/subfind_density.c/96
  77 2990 0             DM_Vz    126.8385   27724.2374  subfind_density()/subfind_density.c/97
  50 2984 0      DataNodeList     58.1818   27073.1102  subfind_density()/subfind_density.c/89
  50 2985 0              Left    126.7926   27199.9028  subfind_density()/subfind_density.c/91
  50 2986 0             Right    126.7926   27326.6955  subfind_density()/subfind_density.c/92
  77 2991 0    	  DensDataGet    116.3701   27840.6075  subfind_density()/subfind_density.c/161
  77 2992 0  	  DensDataResult     87.2776   27927.8851  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  50 2987 0              Todo     15.8491   27342.5446  subfind_density()/subfind_density.c/93
  50 2988 0             DM_Vx    126.7926   27469.3372  subfind_density()/subfind_density.c/95

Task=77: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.1149 MB).
  50 2989 0             DM_Vy    126.7926   27596.1298  subfind_density()/subfind_density.c/96
task 77: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  50 2990 0             DM_Vz    126.7926   27722.9225  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 77
 118 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 118 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 118 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 118 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 118 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 118 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 118 2973 0         FOF_PList    380.5740   18861.3757  fof_fof()/fof.c/249
 118 2974 0         FOF_GList    634.2899   19495.6657  fof_fof()/fof.c/337
 118 2975 0             Group     12.1108   19507.7765  fof_fof()/fof.c/380
 118 2976 0   DomainNodeIndex      0.3461   19508.1227  force_treeallocate()/forcetree.c/4800
   9 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   9 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   9 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   9 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   9 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   9 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
   9 2973 0         FOF_PList    380.5483   18861.3500  fof_fof()/fof.c/249
   9 2974 0         FOF_GList    634.2471   19495.5971  fof_fof()/fof.c/337
   9 2975 0             Group     12.7365   19508.3336  fof_fof()/fof.c/380
   9 2976 0   DomainNodeIndex      0.3461   19508.6798  force_treeallocate()/forcetree.c/4800
   9 2977 0        Nodes_base   3737.5076   23246.1874  force_treeallocate()/forcetree.c/4803
   9 2978 0     Extnodes_base   3114.5897   26360.7771  force_treeallocate()/forcetree.c/4811
   9 2979 0          Nextnode    222.3012   26583.0783  force_treeallocate()/forcetree.c/4820
   9 2980 0            Father    221.9056   26804.9838  force_treeallocate()/forcetree.c/4827
   9 2981 0           Ngblist     63.4247   26868.4086  subfind_density()/subfind_density.c/78
   9 2982 0         Dist2list    126.8494   26995.2580  subfind_density()/subfind_density.c/79
   9 2983 0    DataIndexTable     21.8182   27017.0762  subfind_density()/subfind_density.c/87
   9 2984 0      DataNodeList     58.1818   27075.2579  subfind_density()/subfind_density.c/89
  50 2991 0    	  DensDataGet    122.8365   27845.7590  subfind_density()/subfind_density.c/161
 118 2977 0        Nodes_base   3737.5076   23245.6303  force_treeallocate()/forcetree.c/4803
 118 2978 0     Extnodes_base   3114.5897   26360.2200  force_treeallocate()/forcetree.c/4811
 118 2979 0          Nextnode    222.3012   26582.5211  force_treeallocate()/forcetree.c/4820
 118 2980 0            Father    221.9056   26804.4267  force_treeallocate()/forcetree.c/4827
   9 2985 0              Left    126.8494   27202.1074  subfind_density()/subfind_density.c/91
   9 2986 0             Right    126.8494   27328.9568  subfind_density()/subfind_density.c/92
   9 2987 0              Todo     15.8562   27344.8130  subfind_density()/subfind_density.c/93
   9 2988 0             DM_Vx    126.8494   27471.6624  subfind_density()/subfind_density.c/95
   9 2989 0             DM_Vy    126.8494   27598.5118  subfind_density()/subfind_density.c/96
   9 2990 0             DM_Vz    126.8494   27725.3612  subfind_density()/subfind_density.c/97
   9 2991 0    	  DensDataGet    118.8062   27844.1674  subfind_density()/subfind_density.c/161
   9 2992 0  	  DensDataResult     89.1046   27933.2720  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=9: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.728 MB).
  50 2992 0  	  DensDataResult     92.1274   27937.8864  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 118 2981 0           Ngblist     63.4290   26867.8557  subfind_density()/subfind_density.c/78
 118 2982 0         Dist2list    126.8580   26994.7137  subfind_density()/subfind_density.c/79
 118 2983 0    DataIndexTable     21.8182   27016.5319  subfind_density()/subfind_density.c/87
task 9: endrun called with an error level of 812



Task=50: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.1136 MB).
task 50: endrun called with an error level of 812


 118 2984 0      DataNodeList     58.1818   27074.7137  subfind_density()/subfind_density.c/89
 118 2985 0              Left    126.8580   27201.5717  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 9
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 50
 118 2986 0             Right    126.8580   27328.4296  subfind_density()/subfind_density.c/92
 118 2987 0              Todo     15.8573   27344.2869  subfind_density()/subfind_density.c/93
 118 2988 0             DM_Vx    126.8580   27471.1449  subfind_density()/subfind_density.c/95
 118 2989 0             DM_Vy    126.8580   27598.0029  subfind_density()/subfind_density.c/96
 118 2990 0             DM_Vz    126.8580   27724.8609  subfind_density()/subfind_density.c/97
 118 2991 0    	  DensDataGet    125.7003   27850.5611  subfind_density()/subfind_density.c/161
 118 2992 0  	  DensDataResult     94.2752   27944.8363  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=118: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.1637 MB).
task 118: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 118
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 114 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 114 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 114 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 114 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 114 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 114 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 114 2973 0         FOF_PList    378.9886   18859.7904  fof_fof()/fof.c/249
 114 2974 0         FOF_GList    631.6477   19491.4381  fof_fof()/fof.c/337
 114 2975 0             Group     10.7796   19502.2177  fof_fof()/fof.c/380
  13 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  13 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  13 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  13 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  13 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  13 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  13 2973 0         FOF_PList    380.4252   18861.2270  fof_fof()/fof.c/249
  13 2974 0         FOF_GList    634.0421   19495.2691  fof_fof()/fof.c/337
  13 2975 0             Group     11.6045   19506.8736  fof_fof()/fof.c/380
  13 2976 0   DomainNodeIndex      0.3461   19507.2197  force_treeallocate()/forcetree.c/4800
  13 2977 0        Nodes_base   3737.5076   23244.7273  force_treeallocate()/forcetree.c/4803
  13 2978 0     Extnodes_base   3114.5897   26359.3170  force_treeallocate()/forcetree.c/4811
  13 2979 0          Nextnode    222.3012   26581.6182  force_treeallocate()/forcetree.c/4820
  13 2980 0            Father    221.9056   26803.5238  force_treeallocate()/forcetree.c/4827
  13 2981 0           Ngblist     63.4042   26866.9280  subfind_density()/subfind_density.c/78
  13 2982 0         Dist2list    126.8084   26993.7364  subfind_density()/subfind_density.c/79
  13 2983 0    DataIndexTable     21.8182   27015.5546  subfind_density()/subfind_density.c/87
  13 2984 0      DataNodeList     58.1818   27073.7364  subfind_density()/subfind_density.c/89
 114 2976 0   DomainNodeIndex      0.3461   19502.5639  force_treeallocate()/forcetree.c/4800
 114 2977 0        Nodes_base   3737.5076   23240.0715  force_treeallocate()/forcetree.c/4803
  13 2985 0              Left    126.8084   27200.5448  subfind_density()/subfind_density.c/91
  13 2986 0             Right    126.8084   27327.3532  subfind_density()/subfind_density.c/92
  13 2987 0              Todo     15.8511   27343.2042  subfind_density()/subfind_density.c/93
  13 2988 0             DM_Vx    126.8084   27470.0126  subfind_density()/subfind_density.c/95
  13 2989 0             DM_Vy    126.8084   27596.8210  subfind_density()/subfind_density.c/96
  13 2990 0             DM_Vz    126.8084   27723.6295  subfind_density()/subfind_density.c/97
  13 2991 0    	  DensDataGet    118.4278   27842.0573  subfind_density()/subfind_density.c/161
  13 2992 0  	  DensDataResult     88.8208   27930.8781  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 114 2978 0     Extnodes_base   3114.5897   26354.6612  force_treeallocate()/forcetree.c/4811
 114 2979 0          Nextnode    222.3012   26576.9624  force_treeallocate()/forcetree.c/4820
 114 2980 0            Father    221.9056   26798.8679  force_treeallocate()/forcetree.c/4827

Task=13: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.1219 MB).
task 13: endrun called with an error level of 812


 114 2981 0           Ngblist     63.1648   26862.0327  subfind_density()/subfind_density.c/78
 114 2982 0         Dist2list    126.3295   26988.3623  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 13
 114 2983 0    DataIndexTable     21.8182   27010.1804  subfind_density()/subfind_density.c/87
 114 2984 0      DataNodeList     58.1818   27068.3622  subfind_density()/subfind_density.c/89
 114 2985 0              Left    126.3295   27194.6918  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 114 2986 0             Right    126.3295   27321.0213  subfind_density()/subfind_density.c/92
  76 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  76 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  76 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 114 2987 0              Todo     15.7912   27336.8125  subfind_density()/subfind_density.c/93
  76 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  76 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 114 2988 0             DM_Vx    126.3295   27463.1421  subfind_density()/subfind_density.c/95
  76 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  76 2973 0         FOF_PList    380.0727   18860.8745  fof_fof()/fof.c/249
  76 2974 0         FOF_GList    633.4545   19494.3289  fof_fof()/fof.c/337
 114 2989 0             DM_Vy    126.3295   27589.4716  subfind_density()/subfind_density.c/96
  76 2975 0             Group     11.3264   19505.6554  fof_fof()/fof.c/380
  76 2976 0   DomainNodeIndex      0.3461   19506.0015  force_treeallocate()/forcetree.c/4800
 114 2990 0             DM_Vz    126.3295   27715.8011  subfind_density()/subfind_density.c/97
  76 2977 0        Nodes_base   3737.5076   23243.5091  force_treeallocate()/forcetree.c/4803
  76 2978 0     Extnodes_base   3114.5897   26358.0988  force_treeallocate()/forcetree.c/4811
 114 2991 0    	  DensDataGet    126.9985   27842.7996  subfind_density()/subfind_density.c/161
  76 2979 0          Nextnode    222.3012   26580.4000  force_treeallocate()/forcetree.c/4820
  76 2980 0            Father    221.9056   26802.3056  force_treeallocate()/forcetree.c/4827
 114 2992 0  	  DensDataResult     95.2489   27938.0485  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  76 2981 0           Ngblist     63.3455   26865.6510  subfind_density()/subfind_density.c/78
  76 2982 0         Dist2list    126.6909   26992.3419  subfind_density()/subfind_density.c/79

Task=114: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.9515 MB).
  76 2983 0    DataIndexTable     21.8182   27014.1601  subfind_density()/subfind_density.c/87
task 114: endrun called with an error level of 812


  76 2984 0      DataNodeList     58.1818   27072.3419  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 114
  76 2985 0              Left    126.6909   27199.0328  subfind_density()/subfind_density.c/91
  76 2986 0             Right    126.6909   27325.7237  subfind_density()/subfind_density.c/92
  76 2987 0              Todo     15.8364   27341.5600  subfind_density()/subfind_density.c/93
  76 2988 0             DM_Vx    126.6909   27468.2509  subfind_density()/subfind_density.c/95
  76 2989 0             DM_Vy    126.6909   27594.9418  subfind_density()/subfind_density.c/96
  76 2990 0             DM_Vz    126.6909   27721.6327  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  76 2991 0    	  DensDataGet    123.6177   27845.2504  subfind_density()/subfind_density.c/161
  58 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  58 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  58 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  58 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  76 2992 0  	  DensDataResult     92.7133   27937.9637  subfind_density()/subfind_density.c/201
  58 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  58 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  58 2973 0         FOF_PList    380.2607   18861.0625  fof_fof()/fof.c/249
----------------------------------------------------------------------------------------
  58 2974 0         FOF_GList    633.7678   19494.8303  fof_fof()/fof.c/337
  58 2975 0             Group     12.9254   19507.7556  fof_fof()/fof.c/380
  58 2976 0   DomainNodeIndex      0.3461   19508.1018  force_treeallocate()/forcetree.c/4800

Task=76: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.0363 MB).
task 76: endrun called with an error level of 812


  58 2977 0        Nodes_base   3737.5076   23245.6094  force_treeallocate()/forcetree.c/4803
  58 2978 0     Extnodes_base   3114.5897   26360.1991  force_treeallocate()/forcetree.c/4811
  58 2979 0          Nextnode    222.3012   26582.5003  force_treeallocate()/forcetree.c/4820
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 76
  58 2980 0            Father    221.9056   26804.4058  force_treeallocate()/forcetree.c/4827
  58 2981 0           Ngblist     63.3768   26867.7826  subfind_density()/subfind_density.c/78
  58 2982 0         Dist2list    126.7536   26994.5362  subfind_density()/subfind_density.c/79
  58 2983 0    DataIndexTable     21.8182   27016.3544  subfind_density()/subfind_density.c/87
  58 2984 0      DataNodeList     58.1818   27074.5362  subfind_density()/subfind_density.c/89
  58 2985 0              Left    126.7536   27201.2897  subfind_density()/subfind_density.c/91
  58 2986 0             Right    126.7536   27328.0433  subfind_density()/subfind_density.c/92
  58 2987 0              Todo     15.8442   27343.8875  subfind_density()/subfind_density.c/93
  58 2988 0             DM_Vx    126.7536   27470.6411  subfind_density()/subfind_density.c/95
  58 2989 0             DM_Vy    126.7536   27597.3946  subfind_density()/subfind_density.c/96
  58 2990 0             DM_Vz    126.7536   27724.1482  subfind_density()/subfind_density.c/97
  58 2991 0    	  DensDataGet    110.4989   27834.6471  subfind_density()/subfind_density.c/161
  58 2992 0  	  DensDataResult     82.8742   27917.5213  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=58: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.4787 MB).
task 58: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 58
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  12 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  12 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  12 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  12 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  12 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  12 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  12 2973 0         FOF_PList    380.5103   18861.3121  fof_fof()/fof.c/249
  12 2974 0         FOF_GList    634.1838   19495.4959  fof_fof()/fof.c/337
  12 2975 0             Group     10.3220   19505.8180  fof_fof()/fof.c/380
 115 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 115 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 115 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 115 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 115 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 115 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 115 2973 0         FOF_PList    380.4426   18861.2444  fof_fof()/fof.c/249
 115 2974 0         FOF_GList    634.0711   19495.3155  fof_fof()/fof.c/337
 115 2975 0             Group     13.6309   19508.9464  fof_fof()/fof.c/380
 115 2976 0   DomainNodeIndex      0.3461   19509.2926  force_treeallocate()/forcetree.c/4800
 115 2977 0        Nodes_base   3737.5076   23246.8002  force_treeallocate()/forcetree.c/4803
 115 2978 0     Extnodes_base   3114.5897   26361.3899  force_treeallocate()/forcetree.c/4811
 115 2979 0          Nextnode    222.3012   26583.6911  force_treeallocate()/forcetree.c/4820
  12 2976 0   DomainNodeIndex      0.3461   19506.1641  force_treeallocate()/forcetree.c/4800
  12 2977 0        Nodes_base   3737.5076   23243.6717  force_treeallocate()/forcetree.c/4803
  12 2978 0     Extnodes_base   3114.5897   26358.2614  force_treeallocate()/forcetree.c/4811
  12 2979 0          Nextnode    222.3012   26580.5626  force_treeallocate()/forcetree.c/4820
 115 2980 0            Father    221.9056   26805.5966  force_treeallocate()/forcetree.c/4827
 115 2981 0           Ngblist     63.4071   26869.0038  subfind_density()/subfind_density.c/78
 115 2982 0         Dist2list    126.8142   26995.8180  subfind_density()/subfind_density.c/79
 115 2983 0    DataIndexTable     21.8182   27017.6361  subfind_density()/subfind_density.c/87
  12 2980 0            Father    221.9056   26802.4682  force_treeallocate()/forcetree.c/4827
  12 2981 0           Ngblist     63.4184   26865.8866  subfind_density()/subfind_density.c/78
  12 2982 0         Dist2list    126.8368   26992.7233  subfind_density()/subfind_density.c/79
 115 2984 0      DataNodeList     58.1818   27075.8179  subfind_density()/subfind_density.c/89
 115 2985 0              Left    126.8142   27202.6322  subfind_density()/subfind_density.c/91
 115 2986 0             Right    126.8142   27329.4464  subfind_density()/subfind_density.c/92
  12 2983 0    DataIndexTable     21.8182   27014.5415  subfind_density()/subfind_density.c/87
  12 2984 0      DataNodeList     58.1818   27072.7233  subfind_density()/subfind_density.c/89
  12 2985 0              Left    126.8368   27199.5601  subfind_density()/subfind_density.c/91
 115 2987 0              Todo     15.8518   27345.2982  subfind_density()/subfind_density.c/93
 115 2988 0             DM_Vx    126.8142   27472.1124  subfind_density()/subfind_density.c/95
 115 2989 0             DM_Vy    126.8142   27598.9266  subfind_density()/subfind_density.c/96
  12 2986 0             Right    126.8368   27326.3968  subfind_density()/subfind_density.c/92
  12 2987 0              Todo     15.8546   27342.2514  subfind_density()/subfind_density.c/93
  12 2988 0             DM_Vx    126.8368   27469.0882  subfind_density()/subfind_density.c/95
 115 2990 0             DM_Vz    126.8142   27725.7408  subfind_density()/subfind_density.c/97
 115 2991 0    	  DensDataGet    113.6371   27839.3780  subfind_density()/subfind_density.c/161
 115 2992 0  	  DensDataResult     85.2279   27924.6058  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  12 2989 0             DM_Vy    126.8368   27595.9250  subfind_density()/subfind_density.c/96
  12 2990 0             DM_Vz    126.8368   27722.7617  subfind_density()/subfind_density.c/97

Task=115: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.3942 MB).
task 115: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
   1 2973 0         FOF_PList    380.3405   18861.1423  fof_fof()/fof.c/249
  12 2991 0    	  DensDataGet    108.6823   27831.4440  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 115
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2974 0         FOF_GList    633.9008   19495.0431  fof_fof()/fof.c/337
   1 2975 0             Group     10.5959   19505.6390  fof_fof()/fof.c/380
   1 2976 0   DomainNodeIndex      0.3461   19505.9851  force_treeallocate()/forcetree.c/4800
   1 2977 0        Nodes_base   3737.5076   23243.4928  force_treeallocate()/forcetree.c/4803
   1 2978 0     Extnodes_base   3114.5897   26358.0825  force_treeallocate()/forcetree.c/4811
   1 2979 0          Nextnode    222.3012   26580.3836  force_treeallocate()/forcetree.c/4820
   1 2980 0            Father    221.9056   26802.2892  force_treeallocate()/forcetree.c/4827
   1 2981 0           Ngblist     63.3901   26865.6793  subfind_density()/subfind_density.c/78
   1 2982 0         Dist2list    126.7802   26992.4594  subfind_density()/subfind_density.c/79
  12 2992 0  	  DensDataResult     81.5117   27912.9558  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 126 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 126 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 126 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 126 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 126 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 126 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 126 2973 0         FOF_PList    380.5307   18861.3324  fof_fof()/fof.c/249
 126 2974 0         FOF_GList    634.2178   19495.5502  fof_fof()/fof.c/337
 126 2975 0             Group     10.3220   19505.8722  fof_fof()/fof.c/380
 126 2976 0   DomainNodeIndex      0.3461   19506.2184  force_treeallocate()/forcetree.c/4800
 126 2977 0        Nodes_base   3737.5076   23243.7260  force_treeallocate()/forcetree.c/4803
 126 2978 0     Extnodes_base   3114.5897   26358.3157  force_treeallocate()/forcetree.c/4811
 126 2979 0          Nextnode    222.3012   26580.6168  force_treeallocate()/forcetree.c/4820
   1 2983 0    DataIndexTable     21.8182   27014.2776  subfind_density()/subfind_density.c/87
   1 2984 0      DataNodeList     58.1818   27072.4594  subfind_density()/subfind_density.c/89

Task=12: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=87.0442 MB).
task 12: endrun called with an error level of 812


 126 2980 0            Father    221.9056   26802.5224  force_treeallocate()/forcetree.c/4827
 126 2981 0           Ngblist     63.4218   26865.9442  subfind_density()/subfind_density.c/78
 126 2982 0         Dist2list    126.8436   26992.7878  subfind_density()/subfind_density.c/79
 126 2983 0    DataIndexTable     21.8182   27014.6059  subfind_density()/subfind_density.c/87
   1 2985 0              Left    126.7802   27199.2396  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 12
 126 2984 0      DataNodeList     58.1818   27072.7877  subfind_density()/subfind_density.c/89
 126 2985 0              Left    126.8436   27199.6313  subfind_density()/subfind_density.c/91
 126 2986 0             Right    126.8436   27326.4748  subfind_density()/subfind_density.c/92
   1 2986 0             Right    126.7802   27326.0197  subfind_density()/subfind_density.c/92
 126 2987 0              Todo     15.8554   27342.3303  subfind_density()/subfind_density.c/93
 126 2988 0             DM_Vx    126.8436   27469.1738  subfind_density()/subfind_density.c/95
 126 2989 0             DM_Vy    126.8436   27596.0174  subfind_density()/subfind_density.c/96
   1 2987 0              Todo     15.8475   27341.8673  subfind_density()/subfind_density.c/93
   1 2988 0             DM_Vx    126.7802   27468.6474  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 113 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 113 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 126 2990 0             DM_Vz    126.8436   27722.8609  subfind_density()/subfind_density.c/97
 126 2991 0    	  DensDataGet    119.2111   27842.0720  subfind_density()/subfind_density.c/161
 126 2992 0  	  DensDataResult     89.4083   27931.4803  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   1 2989 0             DM_Vy    126.7802   27595.4276  subfind_density()/subfind_density.c/96
 113 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 113 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 113 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 113 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 113 2973 0         FOF_PList    380.5150   18861.3168  fof_fof()/fof.c/249
 113 2974 0         FOF_GList    634.1917   19495.5086  fof_fof()/fof.c/337
 113 2975 0             Group     10.6634   19506.1720  fof_fof()/fof.c/380
 113 2976 0   DomainNodeIndex      0.3461   19506.5181  force_treeallocate()/forcetree.c/4800
 113 2977 0        Nodes_base   3737.5076   23244.0257  force_treeallocate()/forcetree.c/4803
 113 2978 0     Extnodes_base   3114.5897   26358.6154  force_treeallocate()/forcetree.c/4811
 113 2979 0          Nextnode    222.3012   26580.9166  force_treeallocate()/forcetree.c/4820
 113 2980 0            Father    221.9056   26802.8222  force_treeallocate()/forcetree.c/4827
 113 2981 0           Ngblist     63.4192   26866.2413  subfind_density()/subfind_density.c/78
 113 2982 0         Dist2list    126.8383   26993.0797  subfind_density()/subfind_density.c/79
 113 2983 0    DataIndexTable     21.8182   27014.8979  subfind_density()/subfind_density.c/87

Task=126: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.5197 MB).
task 126: endrun called with an error level of 812


   1 2990 0             DM_Vz    126.7802   27722.2077  subfind_density()/subfind_density.c/97
 113 2984 0      DataNodeList     58.1818   27073.0797  subfind_density()/subfind_density.c/89
 113 2985 0              Left    126.8383   27199.9180  subfind_density()/subfind_density.c/91
 113 2986 0             Right    126.8383   27326.7564  subfind_density()/subfind_density.c/92
 113 2987 0              Todo     15.8548   27342.6112  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 126
   1 2991 0    	  DensDataGet    128.4641   27850.6719  subfind_density()/subfind_density.c/161
   1 2992 0  	  DensDataResult     96.3481   27947.0199  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------- 113 2988 0             DM_Vx    126.8383   27469.4495  subfind_density()/subfind_density.c/95
 113 2989 0             DM_Vy    126.8383   27596.2878  subfind_density()/subfind_density.c/96
 113 2990 0             DM_Vz    126.8383   27723.1262  subfind_density()/subfind_density.c/97

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 113 2991 0    	  DensDataGet    134.8942   27858.0204  subfind_density()/subfind_density.c/161
 113 2992 0  	  DensDataResult    101.1706   27959.1910  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=113: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=40.809 MB).
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=1: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.9801 MB).
task 1: endrun called with an error level of 812


   2 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   2 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   2 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   2 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   2 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   2 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
   2 2973 0         FOF_PList    380.3963   18861.1980  fof_fof()/fof.c/249
   2 2974 0         FOF_GList    633.9938   19495.1918  fof_fof()/fof.c/337
   2 2975 0             Group     11.3658   19506.5576  fof_fof()/fof.c/380
   2 2976 0   DomainNodeIndex      0.3461   19506.9038  force_treeallocate()/forcetree.c/4800
   2 2977 0        Nodes_base   3737.5076   23244.4114  force_treeallocate()/forcetree.c/4803
task 113: endrun called with an error level of 812


 125 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 125 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 125 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 125 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 125 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 125 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 125 2973 0         FOF_PList    380.5759   18861.3776  fof_fof()/fof.c/249
 125 2974 0         FOF_GList    634.2931   19495.6707  fof_fof()/fof.c/337
 125 2975 0             Group     10.3646   19506.0353  fof_fof()/fof.c/380
 125 2976 0   DomainNodeIndex      0.3461   19506.3814  force_treeallocate()/forcetree.c/4800
 125 2977 0        Nodes_base   3737.5076   23243.8891  force_treeallocate()/forcetree.c/4803
 125 2978 0     Extnodes_base   3114.5897   26358.4788  force_treeallocate()/forcetree.c/4811
 125 2979 0          Nextnode    222.3012   26580.7799  force_treeallocate()/forcetree.c/4820
 125 2980 0            Father    221.9056   26802.6855  force_treeallocate()/forcetree.c/4827
 125 2981 0           Ngblist     63.4293   26866.1148  subfind_density()/subfind_density.c/78
 125 2982 0         Dist2list    126.8586   26992.9734  subfind_density()/subfind_density.c/79
 125 2983 0    DataIndexTable     21.8182   27014.7916  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 1
   2 2978 0     Extnodes_base   3114.5897   26359.0011  force_treeallocate()/forcetree.c/4811
   2 2979 0          Nextnode    222.3012   26581.3023  force_treeallocate()/forcetree.c/4820
   2 2980 0            Father    221.9056   26803.2078  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 113
 125 2984 0      DataNodeList     58.1818   27072.9734  subfind_density()/subfind_density.c/89
 125 2985 0              Left    126.8586   27199.8320  subfind_density()/subfind_density.c/91
 125 2986 0             Right    126.8586   27326.6907  subfind_density()/subfind_density.c/92
 125 2987 0              Todo     15.8573   27342.5480  subfind_density()/subfind_density.c/93
 125 2988 0             DM_Vx    126.8586   27469.4066  subfind_density()/subfind_density.c/95
 125 2989 0             DM_Vy    126.8586   27596.2652  subfind_density()/subfind_density.c/96
 125 2990 0             DM_Vz    126.8586   27723.1238  subfind_density()/subfind_density.c/97
   2 2981 0           Ngblist     63.3994   26866.6072  subfind_density()/subfind_density.c/78
   2 2982 0         Dist2list    126.7988   26993.4060  subfind_density()/subfind_density.c/79
   2 2983 0    DataIndexTable     21.8182   27015.2242  subfind_density()/subfind_density.c/87
 125 2991 0    	  DensDataGet    143.9221   27867.0460  subfind_density()/subfind_density.c/161
 125 2992 0  	  DensDataResult    107.9416   27974.9875  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=125: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=25.0125 MB).
   2 2984 0      DataNodeList     58.1818   27073.4059  subfind_density()/subfind_density.c/89
   2 2985 0              Left    126.7988   27200.2047  subfind_density()/subfind_density.c/91
   2 2986 0             Right    126.7988   27327.0034  subfind_density()/subfind_density.c/92
   2 2987 0              Todo     15.8498   27342.8533  subfind_density()/subfind_density.c/93
task 125: endrun called with an error level of 812


   2 2988 0             DM_Vx    126.7988   27469.6520  subfind_density()/subfind_density.c/95
   2 2989 0             DM_Vy    126.7988   27596.4508  subfind_density()/subfind_density.c/96
   2 2990 0             DM_Vz    126.7988   27723.2495  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 125
   2 2991 0    	  DensDataGet    109.1935   27832.4430  subfind_density()/subfind_density.c/161
   2 2992 0  	  DensDataResult     81.8951   27914.3381  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=2: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.6619 MB).
task 2: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 2
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 112 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 112 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 112 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 112 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 112 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 112 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 112 2973 0         FOF_PList    380.5264   18861.3281  fof_fof()/fof.c/249
  15 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  15 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  15 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  15 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  15 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  15 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  15 2973 0         FOF_PList    380.4254   18861.2272  fof_fof()/fof.c/249
  15 2974 0         FOF_GList    634.0423   19495.2694  fof_fof()/fof.c/337
  15 2975 0             Group     11.9957   19507.2651  fof_fof()/fof.c/380
  15 2976 0   DomainNodeIndex      0.3461   19507.6112  force_treeallocate()/forcetree.c/4800
  15 2977 0        Nodes_base   3737.5076   23245.1189  force_treeallocate()/forcetree.c/4803
  15 2978 0     Extnodes_base   3114.5897   26359.7086  force_treeallocate()/forcetree.c/4811
  15 2979 0          Nextnode    222.3012   26582.0097  force_treeallocate()/forcetree.c/4820
  15 2980 0            Father    221.9056   26803.9153  force_treeallocate()/forcetree.c/4827
  15 2981 0           Ngblist     63.4042   26867.3195  subfind_density()/subfind_density.c/78
  15 2982 0         Dist2list    126.8085   26994.1280  subfind_density()/subfind_density.c/79
  15 2983 0    DataIndexTable     21.8182   27015.9462  subfind_density()/subfind_density.c/87
 112 2974 0         FOF_GList    634.2106   19495.5387  fof_fof()/fof.c/337
 112 2975 0             Group     11.8151   19507.3539  fof_fof()/fof.c/380
  15 2984 0      DataNodeList     58.1818   27074.1280  subfind_density()/subfind_density.c/89
  15 2985 0              Left    126.8085   27200.9364  subfind_density()/subfind_density.c/91
  15 2986 0             Right    126.8085   27327.7449  subfind_density()/subfind_density.c/92
  15 2987 0              Todo     15.8511   27343.5959  subfind_density()/subfind_density.c/93
  15 2988 0             DM_Vx    126.8085   27470.4044  subfind_density()/subfind_density.c/95
  15 2989 0             DM_Vy    126.8085   27597.2128  subfind_density()/subfind_density.c/96
 112 2976 0   DomainNodeIndex      0.3461   19507.7000  force_treeallocate()/forcetree.c/4800
 112 2977 0        Nodes_base   3737.5076   23245.2076  force_treeallocate()/forcetree.c/4803
 112 2978 0     Extnodes_base   3114.5897   26359.7973  force_treeallocate()/forcetree.c/4811
  15 2990 0             DM_Vz    126.8085   27724.0213  subfind_density()/subfind_density.c/97
  15 2991 0    	  DensDataGet    122.2778   27846.2991  subfind_density()/subfind_density.c/161
  15 2992 0  	  DensDataResult     91.7083   27938.0074  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 112 2979 0          Nextnode    222.3012   26582.0985  force_treeallocate()/forcetree.c/4820
 112 2980 0            Father    221.9056   26804.0041  force_treeallocate()/forcetree.c/4827

Task=15: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.9926 MB).
task 15: endrun called with an error level of 812


 112 2981 0           Ngblist     63.4211   26867.4251  subfind_density()/subfind_density.c/78
 112 2982 0         Dist2list    126.8421   26994.2672  subfind_density()/subfind_density.c/79
 112 2983 0    DataIndexTable     21.8182   27016.0854  subfind_density()/subfind_density.c/87
 112 2984 0      DataNodeList     58.1818   27074.2672  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 15
 112 2985 0              Left    126.8421   27201.1093  subfind_density()/subfind_density.c/91
 112 2986 0             Right    126.8421   27327.9514  subfind_density()/subfind_density.c/92
 112 2987 0              Todo     15.8553   27343.8067  subfind_density()/subfind_density.c/93
 112 2988 0             DM_Vx    126.8421   27470.6488  subfind_density()/subfind_density.c/95
 112 2989 0             DM_Vy    126.8421   27597.4909  subfind_density()/subfind_density.c/96
 112 2990 0             DM_Vz    126.8421   27724.3331  subfind_density()/subfind_density.c/97
 112 2991 0    	  DensDataGet    111.7120   27836.0450  subfind_density()/subfind_density.c/161
 112 2992 0  	  DensDataResult     83.7840   27919.8290  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=112: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.171 MB).
task 112: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 112
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 112task 112: endrun called with an error level of 124


:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 112
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   8 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   8 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   8 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   8 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   8 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   8 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 119 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 119 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 119 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 119 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 119 2973 0         FOF_PList    380.4837   18861.2855  fof_fof()/fof.c/249
 119 2974 0         FOF_GList    634.1395   19495.4250  fof_fof()/fof.c/337
 119 2975 0             Group     11.7684   19507.1935  fof_fof()/fof.c/380
 119 2976 0   DomainNodeIndex      0.3461   19507.5396  force_treeallocate()/forcetree.c/4800
 119 2977 0        Nodes_base   3737.5076   23245.0472  force_treeallocate()/forcetree.c/4803
 119 2978 0     Extnodes_base   3114.5897   26359.6369  force_treeallocate()/forcetree.c/4811
   8 2973 0         FOF_PList    380.4940   18861.2958  fof_fof()/fof.c/249
   8 2974 0         FOF_GList    634.1567   19495.4525  fof_fof()/fof.c/337
   8 2975 0             Group     10.9342   19506.3867  fof_fof()/fof.c/380
 119 2979 0          Nextnode    222.3012   26581.9381  force_treeallocate()/forcetree.c/4820
 119 2980 0            Father    221.9056   26803.8437  force_treeallocate()/forcetree.c/4827
 119 2981 0           Ngblist     63.4140   26867.2576  subfind_density()/subfind_density.c/78
 119 2982 0         Dist2list    126.8279   26994.0855  subfind_density()/subfind_density.c/79
   8 2976 0   DomainNodeIndex      0.3461   19506.7328  force_treeallocate()/forcetree.c/4800
   8 2977 0        Nodes_base   3737.5076   23244.2404  force_treeallocate()/forcetree.c/4803
   8 2978 0     Extnodes_base   3114.5897   26358.8301  force_treeallocate()/forcetree.c/4811
 119 2983 0    DataIndexTable     21.8182   27015.9037  subfind_density()/subfind_density.c/87
 119 2984 0      DataNodeList     58.1818   27074.0855  subfind_density()/subfind_density.c/89
 119 2985 0              Left    126.8279   27200.9134  subfind_density()/subfind_density.c/91
 119 2986 0             Right    126.8279   27327.7413  subfind_density()/subfind_density.c/92
   8 2979 0          Nextnode    222.3012   26581.1313  force_treeallocate()/forcetree.c/4820
   8 2980 0            Father    221.9056   26803.0369  force_treeallocate()/forcetree.c/4827
 119 2987 0              Todo     15.8535   27343.5948  subfind_density()/subfind_density.c/93
 119 2988 0             DM_Vx    126.8279   27470.4227  subfind_density()/subfind_density.c/95
 119 2989 0             DM_Vy    126.8279   27597.2506  subfind_density()/subfind_density.c/96
 119 2990 0             DM_Vz    126.8279   27724.0785  subfind_density()/subfind_density.c/97
 119 2991 0    	  DensDataGet    112.6750   27836.7535  subfind_density()/subfind_density.c/161
 119 2992 0  	  DensDataResult     84.5063   27921.2598  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=119: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.7402 MB).
task 119: endrun called with an error level of 812


   8 2981 0           Ngblist     63.4157   26866.4526  subfind_density()/subfind_density.c/78
   8 2982 0         Dist2list    126.8313   26993.2839  subfind_density()/subfind_density.c/79
   8 2983 0    DataIndexTable     21.8182   27015.1021  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 119
   8 2984 0      DataNodeList     58.1818   27073.2839  subfind_density()/subfind_density.c/89
   8 2985 0              Left    126.8313   27200.1152  subfind_density()/subfind_density.c/91
   8 2986 0             Right    126.8313   27326.9465  subfind_density()/subfind_density.c/92
   8 2987 0              Todo     15.8539   27342.8005  subfind_density()/subfind_density.c/93
   8 2988 0             DM_Vx    126.8313   27469.6318  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   8 2989 0             DM_Vy    126.8313   27596.4631  subfind_density()/subfind_density.c/96
  78 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  78 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  78 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  78 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  78 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  78 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  78 2973 0         FOF_PList    380.4513   18861.2531  fof_fof()/fof.c/249
  78 2974 0         FOF_GList    634.0856   19495.3387  fof_fof()/fof.c/337
   8 2990 0             DM_Vz    126.8313   27723.2945  subfind_density()/subfind_density.c/97
  78 2975 0             Group     11.3866   19506.7253  fof_fof()/fof.c/380
  78 2976 0   DomainNodeIndex      0.3461   19507.0714  force_treeallocate()/forcetree.c/4800
   8 2991 0    	  DensDataGet    110.5245   27833.8189  subfind_density()/subfind_density.c/161
  78 2977 0        Nodes_base   3737.5076   23244.5791  force_treeallocate()/forcetree.c/4803
  78 2978 0     Extnodes_base   3114.5897   26359.1688  force_treeallocate()/forcetree.c/4811
  78 2979 0          Nextnode    222.3012   26581.4699  force_treeallocate()/forcetree.c/4820
   8 2992 0  	  DensDataResult     82.8934   27916.7123  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  78 2980 0            Father    221.9056   26803.3755  force_treeallocate()/forcetree.c/4827
  78 2981 0           Ngblist     63.4086   26866.7841  subfind_density()/subfind_density.c/78
  78 2982 0         Dist2list    126.8171   26993.6012  subfind_density()/subfind_density.c/79

Task=8: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.2877 MB).
task 8: endrun called with an error level of 812


  78 2983 0    DataIndexTable     21.8182   27015.4194  subfind_density()/subfind_density.c/87
  78 2984 0      DataNodeList     58.1818   27073.6012  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 8
  78 2985 0              Left    126.8171   27200.4183  subfind_density()/subfind_density.c/91
  78 2986 0             Right    126.8171   27327.2354  subfind_density()/subfind_density.c/92
task 8: endrun called with an error level of 124


  78 2987 0              Todo     15.8521   27343.0875  subfind_density()/subfind_density.c/93
  78 2988 0             DM_Vx    126.8171   27469.9046  subfind_density()/subfind_density.c/95
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 8:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 8
  78 2989 0             DM_Vy    126.8171   27596.7218  subfind_density()/subfind_density.c/96
  78 2990 0             DM_Vz    126.8171   27723.5389  subfind_density()/subfind_density.c/97
  78 2991 0    	  DensDataGet    109.8606   27833.3995  subfind_density()/subfind_density.c/161
  78 2992 0  	  DensDataResult     82.3954   27915.7949  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=78: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.2051 MB).
task 78: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  54 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  54 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  54 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  54 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  54 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  54 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  54 2973 0         FOF_PList    379.8061   18860.6079  fof_fof()/fof.c/249
  54 2974 0         FOF_GList    633.0101   19493.6180  fof_fof()/fof.c/337
  54 2975 0             Group     11.5173   19505.1353  fof_fof()/fof.c/380
  73 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  73 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  73 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  73 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  73 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  73 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  73 2973 0         FOF_PList    380.5401   18861.3419  fof_fof()/fof.c/249
  73 2974 0         FOF_GList    634.2336   19495.5755  fof_fof()/fof.c/337
  73 2975 0             Group     10.6686   19506.2440  fof_fof()/fof.c/380
  73 2976 0   DomainNodeIndex      0.3461   19506.5902  force_treeallocate()/forcetree.c/4800
  73 2977 0        Nodes_base   3737.5076   23244.0978  force_treeallocate()/forcetree.c/4803
  73 2978 0     Extnodes_base   3114.5897   26358.6875  force_treeallocate()/forcetree.c/4811
  73 2979 0          Nextnode    222.3012   26580.9887  force_treeallocate()/forcetree.c/4820
  73 2980 0            Father    221.9056   26802.8942  force_treeallocate()/forcetree.c/4827
  73 2981 0           Ngblist     63.4234   26866.3176  subfind_density()/subfind_density.c/78
  73 2982 0         Dist2list    126.8467   26993.1643  subfind_density()/subfind_density.c/79
  54 2976 0   DomainNodeIndex      0.3461   19505.4815  force_treeallocate()/forcetree.c/4800
  54 2977 0        Nodes_base   3737.5076   23242.9891  force_treeallocate()/forcetree.c/4803
  54 2978 0     Extnodes_base   3114.5897   26357.5788  force_treeallocate()/forcetree.c/4811
  73 2983 0    DataIndexTable     21.8182   27014.9825  subfind_density()/subfind_density.c/87
  73 2984 0      DataNodeList     58.1818   27073.1643  subfind_density()/subfind_density.c/89
  73 2985 0              Left    126.8467   27200.0110  subfind_density()/subfind_density.c/91
  73 2986 0             Right    126.8467   27326.8577  subfind_density()/subfind_density.c/92
  73 2987 0              Todo     15.8558   27342.7135  subfind_density()/subfind_density.c/93
  54 2979 0          Nextnode    222.3012   26579.8800  force_treeallocate()/forcetree.c/4820
  54 2980 0            Father    221.9056   26801.7855  force_treeallocate()/forcetree.c/4827
  54 2981 0           Ngblist     63.3010   26865.0866  subfind_density()/subfind_density.c/78
  73 2988 0             DM_Vx    126.8467   27469.5603  subfind_density()/subfind_density.c/95
  73 2989 0             DM_Vy    126.8467   27596.4070  subfind_density()/subfind_density.c/96
  73 2990 0             DM_Vz    126.8467   27723.2537  subfind_density()/subfind_density.c/97
  54 2982 0         Dist2list    126.6020   26991.6886  subfind_density()/subfind_density.c/79
  54 2983 0    DataIndexTable     21.8182   27013.5068  subfind_density()/subfind_density.c/87
  54 2984 0      DataNodeList     58.1818   27071.6886  subfind_density()/subfind_density.c/89
  73 2991 0    	  DensDataGet    125.6058   27848.8595  subfind_density()/subfind_density.c/161
  73 2992 0  	  DensDataResult     94.2044   27943.0639  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=73: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.9361 MB).
  54 2985 0              Left    126.6020   27198.2906  subfind_density()/subfind_density.c/91
  54 2986 0             Right    126.6020   27324.8926  subfind_density()/subfind_density.c/92
task 73: endrun called with an error level of 812


  54 2987 0              Todo     15.8253   27340.7179  subfind_density()/subfind_density.c/93
  54 2988 0             DM_Vx    126.6020   27467.3199  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 73
  54 2989 0             DM_Vy    126.6020   27593.9219  subfind_density()/subfind_density.c/96
  54 2990 0             DM_Vz    126.6020   27720.5240  subfind_density()/subfind_density.c/97
  54 2991 0    	  DensDataGet    123.6652   27844.1891  subfind_density()/subfind_density.c/161
  54 2992 0  	  DensDataResult     92.7489   27936.9380  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=54: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=63.062 MB).
task 54: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 54
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 124 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 124 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 124 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 124 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 124 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 124 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
   3 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   3 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   3 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   3 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   3 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   3 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
   3 2973 0         FOF_PList    380.4677   18861.2695  fof_fof()/fof.c/249
   3 2974 0         FOF_GList    634.1128   19495.3823  fof_fof()/fof.c/337
   3 2975 0             Group     12.1793   19507.5616  fof_fof()/fof.c/380
   3 2976 0   DomainNodeIndex      0.3461   19507.9077  force_treeallocate()/forcetree.c/4800
   3 2977 0        Nodes_base   3737.5076   23245.4154  force_treeallocate()/forcetree.c/4803
   3 2978 0     Extnodes_base   3114.5897   26360.0051  force_treeallocate()/forcetree.c/4811
   3 2979 0          Nextnode    222.3012   26582.3062  force_treeallocate()/forcetree.c/4820
 124 2973 0         FOF_PList    380.4202   18861.2220  fof_fof()/fof.c/249
 124 2974 0         FOF_GList    634.0337   19495.2558  fof_fof()/fof.c/337
 124 2975 0             Group     12.5995   19507.8553  fof_fof()/fof.c/380
   3 2980 0            Father    221.9056   26804.2118  force_treeallocate()/forcetree.c/4827
   3 2981 0           Ngblist     63.4113   26867.6231  subfind_density()/subfind_density.c/78
   3 2982 0         Dist2list    126.8226   26994.4457  subfind_density()/subfind_density.c/79
   3 2983 0    DataIndexTable     21.8182   27016.2638  subfind_density()/subfind_density.c/87
 124 2976 0   DomainNodeIndex      0.3461   19508.2014  force_treeallocate()/forcetree.c/4800
 124 2977 0        Nodes_base   3737.5076   23245.7091  force_treeallocate()/forcetree.c/4803
   3 2984 0      DataNodeList     58.1818   27074.4456  subfind_density()/subfind_density.c/89
   3 2985 0              Left    126.8226   27201.2682  subfind_density()/subfind_density.c/91
   3 2986 0             Right    126.8226   27328.0908  subfind_density()/subfind_density.c/92
   3 2987 0              Todo     15.8528   27343.9436  subfind_density()/subfind_density.c/93
 124 2978 0     Extnodes_base   3114.5897   26360.2988  force_treeallocate()/forcetree.c/4811
 124 2979 0          Nextnode    222.3012   26582.5999  force_treeallocate()/forcetree.c/4820
 124 2980 0            Father    221.9056   26804.5055  force_treeallocate()/forcetree.c/4827
   3 2988 0             DM_Vx    126.8226   27470.7661  subfind_density()/subfind_density.c/95
   3 2989 0             DM_Vy    126.8226   27597.5887  subfind_density()/subfind_density.c/96
   3 2990 0             DM_Vz    126.8226   27724.4113  subfind_density()/subfind_density.c/97
 124 2981 0           Ngblist     63.4034   26867.9089  subfind_density()/subfind_density.c/78
 124 2982 0         Dist2list    126.8067   26994.7156  subfind_density()/subfind_density.c/79
 124 2983 0    DataIndexTable     21.8182   27016.5338  subfind_density()/subfind_density.c/87
   3 2991 0    	  DensDataGet    121.6610   27846.0723  subfind_density()/subfind_density.c/161
   3 2992 0  	  DensDataResult     91.2458   27937.3180  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=3: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.682 MB).
 124 2984 0      DataNodeList     58.1818   27074.7156  subfind_density()/subfind_density.c/89
 124 2985 0              Left    126.8067   27201.5224  subfind_density()/subfind_density.c/91
task 3: endrun called with an error level of 812


 124 2986 0             Right    126.8067   27328.3291  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 3
 124 2987 0              Todo     15.8508   27344.1799  subfind_density()/subfind_density.c/93
 124 2988 0             DM_Vx    126.8067   27470.9867  subfind_density()/subfind_density.c/95
 124 2989 0             DM_Vy    126.8067   27597.7934  subfind_density()/subfind_density.c/96
 124 2990 0             DM_Vz    126.8067   27724.6002  subfind_density()/subfind_density.c/97
 124 2991 0    	  DensDataGet    115.6244   27840.2246  subfind_density()/subfind_density.c/161
 124 2992 0  	  DensDataResult     86.7183   27926.9429  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=124: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.0571 MB).
task 124: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 124
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 108 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 108 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 108 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 108 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 108 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 108 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 108 2973 0         FOF_PList    380.5828   18861.3846  fof_fof()/fof.c/249
 108 2974 0         FOF_GList    634.3047   19495.6894  fof_fof()/fof.c/337
 108 2975 0             Group     12.6130   19508.3024  fof_fof()/fof.c/380
 108 2976 0   DomainNodeIndex      0.3461   19508.6485  force_treeallocate()/forcetree.c/4800
 108 2977 0        Nodes_base   3737.5076   23246.1562  force_treeallocate()/forcetree.c/4803
  19 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  19 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  19 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  19 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  19 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  19 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  19 2973 0         FOF_PList    380.4530   18861.2548  fof_fof()/fof.c/249
  19 2974 0         FOF_GList    634.0883   19495.3430  fof_fof()/fof.c/337
  19 2975 0             Group     13.2636   19508.6067  fof_fof()/fof.c/380
  19 2976 0   DomainNodeIndex      0.3461   19508.9528  force_treeallocate()/forcetree.c/4800
  19 2977 0        Nodes_base   3737.5076   23246.4604  force_treeallocate()/forcetree.c/4803
  19 2978 0     Extnodes_base   3114.5897   26361.0501  force_treeallocate()/forcetree.c/4811
  19 2979 0          Nextnode    222.3012   26583.3513  force_treeallocate()/forcetree.c/4820
  19 2980 0            Father    221.9056   26805.2569  force_treeallocate()/forcetree.c/4827
  19 2981 0           Ngblist     63.4088   26868.6657  subfind_density()/subfind_density.c/78
  19 2982 0         Dist2list    126.8177   26995.4833  subfind_density()/subfind_density.c/79
 108 2978 0     Extnodes_base   3114.5897   26360.7458  force_treeallocate()/forcetree.c/4811
 108 2979 0          Nextnode    222.3012   26583.0470  force_treeallocate()/forcetree.c/4820
 108 2980 0            Father    221.9056   26804.9526  force_treeallocate()/forcetree.c/4827
 108 2981 0           Ngblist     63.4305   26868.3831  subfind_density()/subfind_density.c/78
  19 2983 0    DataIndexTable     21.8182   27017.3015  subfind_density()/subfind_density.c/87
  19 2984 0      DataNodeList     58.1818   27075.4833  subfind_density()/subfind_density.c/89
  19 2985 0              Left    126.8177   27202.3010  subfind_density()/subfind_density.c/91
  19 2986 0             Right    126.8177   27329.1186  subfind_density()/subfind_density.c/92
 108 2982 0         Dist2list    126.8609   26995.2440  subfind_density()/subfind_density.c/79
 108 2983 0    DataIndexTable     21.8182   27017.0622  subfind_density()/subfind_density.c/87
 108 2984 0      DataNodeList     58.1818   27075.2440  subfind_density()/subfind_density.c/89
  19 2987 0              Todo     15.8522   27344.9708  subfind_density()/subfind_density.c/93
  19 2988 0             DM_Vx    126.8177   27471.7885  subfind_density()/subfind_density.c/95
  19 2989 0             DM_Vy    126.8177   27598.6062  subfind_density()/subfind_density.c/96
  19 2990 0             DM_Vz    126.8177   27725.4238  subfind_density()/subfind_density.c/97
 108 2985 0              Left    126.8609   27202.1049  subfind_density()/subfind_density.c/91
 108 2986 0             Right    126.8609   27328.9659  subfind_density()/subfind_density.c/92
 108 2987 0              Todo     15.8576   27344.8235  subfind_density()/subfind_density.c/93
  19 2991 0    	  DensDataGet    145.7938   27871.2176  subfind_density()/subfind_density.c/161
  19 2992 0  	  DensDataResult    109.3453   27980.5629  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 108 2988 0             DM_Vx    126.8609   27471.6844  subfind_density()/subfind_density.c/95
 108 2989 0             DM_Vy    126.8609   27598.5454  subfind_density()/subfind_density.c/96
 108 2990 0             DM_Vz    126.8609   27725.4063  subfind_density()/subfind_density.c/97

Task=19: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=19.4371 MB).
task 19: endrun called with an error level of 812


 108 2991 0    	  DensDataGet    131.7787   27857.1851  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 19
 108 2992 0  	  DensDataResult     98.8341   27956.0191  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=108: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=43.9809 MB).
task 108: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 108
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  55 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  55 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  55 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  55 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  55 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  55 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  55 2973 0         FOF_PList    380.3856   18861.1874  fof_fof()/fof.c/249
  55 2974 0         FOF_GList    633.9761   19495.1635  fof_fof()/fof.c/337
  55 2975 0             Group     11.7632   19506.9267  fof_fof()/fof.c/380
  55 2976 0   DomainNodeIndex      0.3461   19507.2729  force_treeallocate()/forcetree.c/4800
  55 2977 0        Nodes_base   3737.5076   23244.7805  force_treeallocate()/forcetree.c/4803
  55 2978 0     Extnodes_base   3114.5897   26359.3702  force_treeallocate()/forcetree.c/4811
  55 2979 0          Nextnode    222.3012   26581.6713  force_treeallocate()/forcetree.c/4820
  55 2980 0            Father    221.9056   26803.5769  force_treeallocate()/forcetree.c/4827
  55 2981 0           Ngblist     63.3976   26866.9745  subfind_density()/subfind_density.c/78
  55 2982 0         Dist2list    126.7952   26993.7697  subfind_density()/subfind_density.c/79
  55 2983 0    DataIndexTable     21.8182   27015.5879  subfind_density()/subfind_density.c/87
  55 2984 0      DataNodeList     58.1818   27073.7697  subfind_density()/subfind_density.c/89
  55 2985 0              Left    126.7952   27200.5649  subfind_density()/subfind_density.c/91
  55 2986 0             Right    126.7952   27327.3601  subfind_density()/subfind_density.c/92
  55 2987 0              Todo     15.8494   27343.2095  subfind_density()/subfind_density.c/93
  55 2988 0             DM_Vx    126.7952   27470.0048  subfind_density()/subfind_density.c/95
  55 2989 0             DM_Vy    126.7952   27596.8000  subfind_density()/subfind_density.c/96
  55 2990 0             DM_Vz    126.7952   27723.5952  subfind_density()/subfind_density.c/97
  55 2991 0    	  DensDataGet    122.2532   27845.8484  subfind_density()/subfind_density.c/161
  55 2992 0  	  DensDataResult     91.6899   27937.5382  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=55: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.4618 MB).
task 55: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 55
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 116 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 116 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 116 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 116 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 116 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 116 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 116 2973 0         FOF_PList    380.5520   18861.3538  fof_fof()/fof.c/249
 116 2974 0         FOF_GList    634.2533   19495.6071  fof_fof()/fof.c/337
 116 2975 0             Group     12.6597   19508.2668  fof_fof()/fof.c/380
 116 2976 0   DomainNodeIndex      0.3461   19508.6129  force_treeallocate()/forcetree.c/4800
 116 2977 0        Nodes_base   3737.5076   23246.1206  force_treeallocate()/forcetree.c/4803
  11 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  11 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  11 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  11 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  11 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  11 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  11 2973 0         FOF_PList    380.5510   18861.3527  fof_fof()/fof.c/249
  11 2974 0         FOF_GList    634.2516   19495.6043  fof_fof()/fof.c/337
  11 2975 0             Group     10.8387   19506.4431  fof_fof()/fof.c/380
  11 2976 0   DomainNodeIndex      0.3461   19506.7892  force_treeallocate()/forcetree.c/4800
  11 2977 0        Nodes_base   3737.5076   23244.2968  force_treeallocate()/forcetree.c/4803
  11 2978 0     Extnodes_base   3114.5897   26358.8865  force_treeallocate()/forcetree.c/4811
 116 2978 0     Extnodes_base   3114.5897   26360.7103  force_treeallocate()/forcetree.c/4811
 116 2979 0          Nextnode    222.3012   26583.0114  force_treeallocate()/forcetree.c/4820
 116 2980 0            Father    221.9056   26804.9170  force_treeallocate()/forcetree.c/4827
  11 2979 0          Nextnode    222.3012   26581.1877  force_treeallocate()/forcetree.c/4820
  11 2980 0            Father    221.9056   26803.0933  force_treeallocate()/forcetree.c/4827
  11 2981 0           Ngblist     63.4252   26866.5184  subfind_density()/subfind_density.c/78
  11 2982 0         Dist2list    126.8503   26993.3688  subfind_density()/subfind_density.c/79
  11 2983 0    DataIndexTable     21.8182   27015.1869  subfind_density()/subfind_density.c/87
 116 2981 0           Ngblist     63.4253   26868.3423  subfind_density()/subfind_density.c/78
 116 2982 0         Dist2list    126.8507   26995.1930  subfind_density()/subfind_density.c/79
 116 2983 0    DataIndexTable     21.8182   27017.0112  subfind_density()/subfind_density.c/87
  11 2984 0      DataNodeList     58.1818   27073.3687  subfind_density()/subfind_density.c/89
  11 2985 0              Left    126.8503   27200.2191  subfind_density()/subfind_density.c/91
  11 2986 0             Right    126.8503   27327.0694  subfind_density()/subfind_density.c/92
 116 2984 0      DataNodeList     58.1818   27075.1930  subfind_density()/subfind_density.c/89
 116 2985 0              Left    126.8507   27202.0436  subfind_density()/subfind_density.c/91
 116 2986 0             Right    126.8507   27328.8943  subfind_density()/subfind_density.c/92
  11 2987 0              Todo     15.8563   27342.9257  subfind_density()/subfind_density.c/93
  11 2988 0             DM_Vx    126.8503   27469.7760  subfind_density()/subfind_density.c/95
  11 2989 0             DM_Vy    126.8503   27596.6263  subfind_density()/subfind_density.c/96
 116 2987 0              Todo     15.8563   27344.7506  subfind_density()/subfind_density.c/93
 116 2988 0             DM_Vx    126.8507   27471.6013  subfind_density()/subfind_density.c/95
 116 2989 0             DM_Vy    126.8507   27598.4520  subfind_density()/subfind_density.c/96
  11 2990 0             DM_Vz    126.8503   27723.4766  subfind_density()/subfind_density.c/97
  11 2991 0    	  DensDataGet    124.4913   27847.9680  subfind_density()/subfind_density.c/161
  11 2992 0  	  DensDataResult     93.3685   27941.3365  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 116 2990 0             DM_Vz    126.8507   27725.3026  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=11: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.6635 MB).
task 11: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 116 2991 0    	  DensDataGet    111.3985   27836.7011  subfind_density()/subfind_density.c/161
 120 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 120 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 120 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 120 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 120 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 120 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 120 2973 0         FOF_PList    380.5155   18861.3173  fof_fof()/fof.c/249
 120 2974 0         FOF_GList    634.1925   19495.5097  fof_fof()/fof.c/337
 120 2975 0             Group     10.8865   19506.3962  fof_fof()/fof.c/380
 120 2976 0   DomainNodeIndex      0.3461   19506.7423  force_treeallocate()/forcetree.c/4800
 120 2977 0        Nodes_base   3737.5076   23244.2500  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 11
   7 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   7 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   7 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   7 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   7 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
   7 2973 0         FOF_PList    380.4035   18861.2053  fof_fof()/fof.c/249
   7 2974 0         FOF_GList    634.0058   19495.2111  fof_fof()/fof.c/337
   7 2975 0             Group     11.8380   19507.0490  fof_fof()/fof.c/380
   7 2976 0   DomainNodeIndex      0.3461   19507.3952  force_treeallocate()/forcetree.c/4800
   7 2977 0        Nodes_base   3737.5076   23244.9028  force_treeallocate()/forcetree.c/4803
   7 2978 0     Extnodes_base   3114.5897   26359.4925  force_treeallocate()/forcetree.c/4811
   7 2979 0          Nextnode    222.3012   26581.7937  force_treeallocate()/forcetree.c/4820
   7 2980 0            Father    221.9056   26803.6992  force_treeallocate()/forcetree.c/4827
   7 2981 0           Ngblist     63.4006   26867.0998  subfind_density()/subfind_density.c/78
   7 2982 0         Dist2list    126.8012   26993.9010  subfind_density()/subfind_density.c/79
 116 2992 0  	  DensDataResult     83.5489   27920.2500  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 120 2978 0     Extnodes_base   3114.5897   26358.8397  force_treeallocate()/forcetree.c/4811
 120 2979 0          Nextnode    222.3012   26581.1408  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   7 2983 0    DataIndexTable     21.8182   27015.7192  subfind_density()/subfind_density.c/87
   7 2984 0      DataNodeList     58.1818   27073.9010  subfind_density()/subfind_density.c/89
   7 2985 0              Left    126.8012   27200.7021  subfind_density()/subfind_density.c/91
   7 2986 0             Right    126.8012   27327.5033  subfind_density()/subfind_density.c/92
   7 2987 0              Todo     15.8502   27343.3534  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 109 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 109 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55

Task=116: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.75 MB).
task 116: endrun called with an error level of 812


 120 2980 0            Father    221.9056   26803.0464  force_treeallocate()/forcetree.c/4827
 120 2981 0           Ngblist     63.4193   26866.4657  subfind_density()/subfind_density.c/78
  18 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  18 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  18 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  18 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  18 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2988 0             DM_Vx    126.8012   27470.1546  subfind_density()/subfind_density.c/95
   7 2989 0             DM_Vy    126.8012   27596.9558  subfind_density()/subfind_density.c/96
   7 2990 0             DM_Vz    126.8012   27723.7569  subfind_density()/subfind_density.c/97
 109 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 109 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 109 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 109 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 109 2973 0         FOF_PList    380.3654   18861.1672  fof_fof()/fof.c/249
 109 2974 0         FOF_GList    633.9423   19495.1095  fof_fof()/fof.c/337
 109 2975 0             Group     10.9757   19506.0852  fof_fof()/fof.c/380
 109 2976 0   DomainNodeIndex      0.3461   19506.4313  force_treeallocate()/forcetree.c/4800
 109 2977 0        Nodes_base   3737.5076   23243.9389  force_treeallocate()/forcetree.c/4803
 109 2978 0     Extnodes_base   3114.5897   26358.5286  force_treeallocate()/forcetree.c/4811
 109 2979 0          Nextnode    222.3012   26580.8298  force_treeallocate()/forcetree.c/4820
 109 2980 0            Father    221.9056   26802.7354  force_treeallocate()/forcetree.c/4827
 109 2981 0           Ngblist     63.3942   26866.1296  subfind_density()/subfind_density.c/78
 109 2982 0         Dist2list    126.7885   26992.9181  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 116
 120 2982 0         Dist2list    126.8385   26993.3042  subfind_density()/subfind_density.c/79
 120 2983 0    DataIndexTable     21.8182   27015.1223  subfind_density()/subfind_density.c/87
  18 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  18 2973 0         FOF_PList    380.4382   18861.2400  fof_fof()/fof.c/249
  18 2974 0         FOF_GList    634.0636   19495.3036  fof_fof()/fof.c/337
  18 2975 0             Group     11.3690   19506.6726  fof_fof()/fof.c/380
  18 2976 0   DomainNodeIndex      0.3461   19507.0187  force_treeallocate()/forcetree.c/4800
   7 2991 0    	  DensDataGet    132.7816   27856.5385  subfind_density()/subfind_density.c/161
   7 2992 0  	  DensDataResult     99.5862   27956.1247  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 109 2983 0    DataIndexTable     21.8182   27014.7362  subfind_density()/subfind_density.c/87
 109 2984 0      DataNodeList     58.1818   27072.9180  subfind_density()/subfind_density.c/89
 109 2985 0              Left    126.7885   27199.7065  subfind_density()/subfind_density.c/91
 109 2986 0             Right    126.7885   27326.4950  subfind_density()/subfind_density.c/92
 120 2984 0      DataNodeList     58.1818   27073.3041  subfind_density()/subfind_density.c/89
 120 2985 0              Left    126.8385   27200.1426  subfind_density()/subfind_density.c/91
 120 2986 0             Right    126.8385   27326.9811  subfind_density()/subfind_density.c/92
  18 2977 0        Nodes_base   3737.5076   23244.5263  force_treeallocate()/forcetree.c/4803
  18 2978 0     Extnodes_base   3114.5897   26359.1160  force_treeallocate()/forcetree.c/4811

Task=7: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=43.8753 MB).
task 7: endrun called with an error level of 812


 109 2987 0              Todo     15.8486   27342.3435  subfind_density()/subfind_density.c/93
 109 2988 0             DM_Vx    126.7885   27469.1320  subfind_density()/subfind_density.c/95
 109 2989 0             DM_Vy    126.7885   27595.9204  subfind_density()/subfind_density.c/96
 109 2990 0             DM_Vz    126.7885   27722.7089  subfind_density()/subfind_density.c/97
 120 2987 0              Todo     15.8548   27342.8359  subfind_density()/subfind_density.c/93
  18 2979 0          Nextnode    222.3012   26581.4172  force_treeallocate()/forcetree.c/4820
  18 2980 0            Father    221.9056   26803.3228  force_treeallocate()/forcetree.c/4827
  18 2981 0           Ngblist     63.4064   26866.7291  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 7
 109 2991 0    	  DensDataGet    125.0501   27847.7590  subfind_density()/subfind_density.c/161
 109 2992 0  	  DensDataResult     93.7876   27941.5466  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 120 2988 0             DM_Vx    126.8385   27469.6744  subfind_density()/subfind_density.c/95
  18 2982 0         Dist2list    126.8127   26993.5419  subfind_density()/subfind_density.c/79
  18 2983 0    DataIndexTable     21.8182   27015.3600  subfind_density()/subfind_density.c/87
  18 2984 0      DataNodeList     58.1818   27073.5418  subfind_density()/subfind_density.c/89
  18 2985 0              Left    126.8127   27200.3546  subfind_density()/subfind_density.c/91

Task=109: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.4534 MB).
task 109: endrun called with an error level of 812


 120 2989 0             DM_Vy    126.8385   27596.5129  subfind_density()/subfind_density.c/96
  18 2986 0             Right    126.8127   27327.1673  subfind_density()/subfind_density.c/92
  18 2987 0              Todo     15.8516   27343.0189  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 109
 120 2990 0             DM_Vz    126.8385   27723.3514  subfind_density()/subfind_density.c/97
  18 2988 0             DM_Vx    126.8127   27469.8316  subfind_density()/subfind_density.c/95
 120 2991 0    	  DensDataGet    129.0690   27852.4204  subfind_density()/subfind_density.c/161
  18 2989 0             DM_Vy    126.8127   27596.6443  subfind_density()/subfind_density.c/96
 120 2992 0  	  DensDataResult     96.8017   27949.2221  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  18 2990 0             DM_Vz    126.8127   27723.4571  subfind_density()/subfind_density.c/97

Task=120: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.7779 MB).
task 120: endrun called with an error level of 812


  18 2991 0    	  DensDataGet    126.6147   27850.0718  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 120
  18 2992 0  	  DensDataResult     94.9611   27945.0329  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 120: endrun called with an error level of 124



Task=18: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.9671 MB).
task 18: endrun called with an error level of 812


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 120:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 120
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 18
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  17 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  17 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  17 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  17 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  17 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  17 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  17 2973 0         FOF_PList    380.4944   18861.2962  fof_fof()/fof.c/249
  17 2974 0         FOF_GList    634.1573   19495.4535  fof_fof()/fof.c/337
  17 2975 0             Group     12.1295   19507.5830  fof_fof()/fof.c/380
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 111 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 111 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 111 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 111 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 111 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 111 2973 0         FOF_PList    380.5747   18861.3765  fof_fof()/fof.c/249
 111 2974 0         FOF_GList    634.2912   19495.6678  fof_fof()/fof.c/337
 111 2975 0             Group     11.7923   19507.4601  fof_fof()/fof.c/380
 111 2976 0   DomainNodeIndex      0.3461   19507.8062  force_treeallocate()/forcetree.c/4800
 111 2977 0        Nodes_base   3737.5076   23245.3138  force_treeallocate()/forcetree.c/4803
 111 2978 0     Extnodes_base   3114.5897   26359.9035  force_treeallocate()/forcetree.c/4811
 111 2979 0          Nextnode    222.3012   26582.2047  force_treeallocate()/forcetree.c/4820
 111 2980 0            Father    221.9056   26804.1103  force_treeallocate()/forcetree.c/4827
 111 2981 0           Ngblist     63.4291   26867.5394  subfind_density()/subfind_density.c/78
 111 2982 0         Dist2list    126.8582   26994.3976  subfind_density()/subfind_density.c/79
  16 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  16 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  16 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  16 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  16 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  16 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  16 2973 0         FOF_PList    380.4941   18861.2958  fof_fof()/fof.c/249
  16 2974 0         FOF_GList    634.1568   19495.4526  fof_fof()/fof.c/337
  16 2975 0             Group     11.0380   19506.4906  fof_fof()/fof.c/380
  16 2976 0   DomainNodeIndex      0.3461   19506.8367  force_treeallocate()/forcetree.c/4800
  16 2977 0        Nodes_base   3737.5076   23244.3443  force_treeallocate()/forcetree.c/4803
  16 2978 0     Extnodes_base   3114.5897   26358.9340  force_treeallocate()/forcetree.c/4811
  16 2979 0          Nextnode    222.3012   26581.2352  force_treeallocate()/forcetree.c/4820
  16 2980 0            Father    221.9056   26803.1408  force_treeallocate()/forcetree.c/4827
 110 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 110 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 110 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 110 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 110 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 110 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 110 2973 0         FOF_PList    379.7974   18860.5992  fof_fof()/fof.c/249
 110 2974 0         FOF_GList    632.9957   19493.5949  fof_fof()/fof.c/337
 110 2975 0             Group     12.0185   19505.6134  fof_fof()/fof.c/380
 110 2976 0   DomainNodeIndex      0.3461   19505.9596  force_treeallocate()/forcetree.c/4800
 110 2977 0        Nodes_base   3737.5076   23243.4672  force_treeallocate()/forcetree.c/4803
 110 2978 0     Extnodes_base   3114.5897   26358.0569  force_treeallocate()/forcetree.c/4811
 110 2979 0          Nextnode    222.3012   26580.3580  force_treeallocate()/forcetree.c/4820
 110 2980 0            Father    221.9056   26802.2636  force_treeallocate()/forcetree.c/4827
 110 2981 0           Ngblist     63.2996   26865.5632  subfind_density()/subfind_density.c/78
 110 2982 0         Dist2list    126.5991   26992.1624  subfind_density()/subfind_density.c/79
  17 2976 0   DomainNodeIndex      0.3461   19507.9291  force_treeallocate()/forcetree.c/4800
  17 2977 0        Nodes_base   3737.5076   23245.4367  force_treeallocate()/forcetree.c/4803
  17 2978 0     Extnodes_base   3114.5897   26360.0264  force_treeallocate()/forcetree.c/4811
  17 2979 0          Nextnode    222.3012   26582.3276  force_treeallocate()/forcetree.c/4820
  17 2980 0            Father    221.9056   26804.2332  force_treeallocate()/forcetree.c/4827
  17 2981 0           Ngblist     63.4157   26867.6489  subfind_density()/subfind_density.c/78
  17 2982 0         Dist2list    126.8315   26994.4804  subfind_density()/subfind_density.c/79
  17 2983 0    DataIndexTable     21.8182   27016.2985  subfind_density()/subfind_density.c/87
 111 2983 0    DataIndexTable     21.8182   27016.2158  subfind_density()/subfind_density.c/87
 111 2984 0      DataNodeList     58.1818   27074.3976  subfind_density()/subfind_density.c/89
 111 2985 0              Left    126.8582   27201.2558  subfind_density()/subfind_density.c/91
 111 2986 0             Right    126.8582   27328.1141  subfind_density()/subfind_density.c/92
 111 2987 0              Todo     15.8573   27343.9714  subfind_density()/subfind_density.c/93
 111 2988 0             DM_Vx    126.8582   27470.8296  subfind_density()/subfind_density.c/95
 111 2989 0             DM_Vy    126.8582   27597.6879  subfind_density()/subfind_density.c/96
 111 2990 0             DM_Vz    126.8582   27724.5461  subfind_density()/subfind_density.c/97
 111 2991 0    	  DensDataGet    109.2571   27833.8032  subfind_density()/subfind_density.c/161
 111 2992 0  	  DensDataResult     81.9429   27915.7461  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=111: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.2539 MB).
task 111: endrun called with an error level of 812


  16 2981 0           Ngblist     63.4157   26866.5564  subfind_density()/subfind_density.c/78
  16 2982 0         Dist2list    126.8314   26993.3878  subfind_density()/subfind_density.c/79
  16 2983 0    DataIndexTable     21.8182   27015.2060  subfind_density()/subfind_density.c/87
  16 2984 0      DataNodeList     58.1818   27073.3878  subfind_density()/subfind_density.c/89
  16 2985 0              Left    126.8314   27200.2191  subfind_density()/subfind_density.c/91
  16 2986 0             Right    126.8314   27327.0505  subfind_density()/subfind_density.c/92
 110 2983 0    DataIndexTable     21.8182   27013.9805  subfind_density()/subfind_density.c/87
 110 2984 0      DataNodeList     58.1818   27072.1623  subfind_density()/subfind_density.c/89
 110 2985 0              Left    126.5991   27198.7615  subfind_density()/subfind_density.c/91
 110 2986 0             Right    126.5991   27325.3606  subfind_density()/subfind_density.c/92
 110 2987 0              Todo     15.8249   27341.1855  subfind_density()/subfind_density.c/93
 110 2988 0             DM_Vx    126.5991   27467.7847  subfind_density()/subfind_density.c/95
  17 2984 0      DataNodeList     58.1818   27074.4803  subfind_density()/subfind_density.c/89
  17 2985 0              Left    126.8315   27201.3118  subfind_density()/subfind_density.c/91
  17 2986 0             Right    126.8315   27328.1433  subfind_density()/subfind_density.c/92
  17 2987 0              Todo     15.8539   27343.9972  subfind_density()/subfind_density.c/93
  17 2988 0             DM_Vx    126.8315   27470.8287  subfind_density()/subfind_density.c/95
  17 2989 0             DM_Vy    126.8315   27597.6601  subfind_density()/subfind_density.c/96
 110 2989 0             DM_Vy    126.5991   27594.3838  subfind_density()/subfind_density.c/96
 110 2990 0             DM_Vz    126.5991   27720.9829  subfind_density()/subfind_density.c/97
 110 2991 0    	  DensDataGet    142.1386   27863.1216  subfind_density()/subfind_density.c/161
  16 2987 0              Todo     15.8539   27342.9044  subfind_density()/subfind_density.c/93
  16 2988 0             DM_Vx    126.8314   27469.7357  subfind_density()/subfind_density.c/95
  16 2989 0             DM_Vy    126.8314   27596.5671  subfind_density()/subfind_density.c/96
  16 2990 0             DM_Vz    126.8314   27723.3985  subfind_density()/subfind_density.c/97
 110 2992 0  	  DensDataResult    106.6040   27969.7255  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  17 2990 0             DM_Vz    126.8315   27724.4916  subfind_density()/subfind_density.c/97
  17 2991 0    	  DensDataGet    116.4460   27840.9376  subfind_density()/subfind_density.c/161

Task=110: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=30.2745 MB).
task 110: endrun called with an error level of 812


  16 2991 0    	  DensDataGet    120.5850   27843.9834  subfind_density()/subfind_density.c/161
  16 2992 0  	  DensDataResult     90.4387   27934.4221  subfind_density()/subfind_density.c/201
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 110
  17 2992 0  	  DensDataResult     87.3345   27928.2720  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 111
----------------------------------------------------------------------------------------

Task=16: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.5779 MB).

Task=17: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.728 MB).
task 17: endrun called with an error level of 812


task 16: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 16
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 17
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 16:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 16: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 16
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  10 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  10 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  10 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  10 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  10 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  10 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  10 2973 0         FOF_PList    380.3880   18861.1898  fof_fof()/fof.c/249
  10 2974 0         FOF_GList    633.9799   19495.1697  fof_fof()/fof.c/337
  10 2975 0             Group     10.3220   19505.4917  fof_fof()/fof.c/380
  10 2976 0   DomainNodeIndex      0.3461   19505.8379  force_treeallocate()/forcetree.c/4800
  10 2977 0        Nodes_base   3737.5076   23243.3455  force_treeallocate()/forcetree.c/4803
  10 2978 0     Extnodes_base   3114.5897   26357.9352  force_treeallocate()/forcetree.c/4811
  10 2979 0          Nextnode    222.3012   26580.2363  force_treeallocate()/forcetree.c/4820
  10 2980 0            Father    221.9056   26802.1419  force_treeallocate()/forcetree.c/4827
  10 2981 0           Ngblist     63.3980   26865.5399  subfind_density()/subfind_density.c/78
  10 2982 0         Dist2list    126.7960   26992.3359  subfind_density()/subfind_density.c/79
  10 2983 0    DataIndexTable     21.8182   27014.1541  subfind_density()/subfind_density.c/87
  10 2984 0      DataNodeList     58.1818   27072.3359  subfind_density()/subfind_density.c/89
  10 2985 0              Left    126.7960   27199.1319  subfind_density()/subfind_density.c/91
  10 2986 0             Right    126.7960   27325.9279  subfind_density()/subfind_density.c/92
  10 2987 0              Todo     15.8495   27341.7774  subfind_density()/subfind_density.c/93
  10 2988 0             DM_Vx    126.7960   27468.5734  subfind_density()/subfind_density.c/95
  10 2989 0             DM_Vy    126.7960   27595.3693  subfind_density()/subfind_density.c/96
  10 2990 0             DM_Vz    126.7960   27722.1653  subfind_density()/subfind_density.c/97
  10 2991 0    	  DensDataGet    136.5840   27858.7494  subfind_density()/subfind_density.c/161
  10 2992 0  	  DensDataResult    102.4380   27961.1874  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=10: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=38.8126 MB).
task 10: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 10
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  23 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  23 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  23 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  23 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  23 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  23 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  23 2973 0         FOF_PList    380.5783   18861.3801  fof_fof()/fof.c/249
  23 2974 0         FOF_GList    634.2972   19495.6773  fof_fof()/fof.c/337
  23 2975 0             Group     11.9884   19507.6657  fof_fof()/fof.c/380
  23 2976 0   DomainNodeIndex      0.3461   19508.0118  force_treeallocate()/forcetree.c/4800
  23 2977 0        Nodes_base   3737.5076   23245.5194  force_treeallocate()/forcetree.c/4803
  23 2978 0     Extnodes_base   3114.5897   26360.1091  force_treeallocate()/forcetree.c/4811
  23 2979 0          Nextnode    222.3012   26582.4103  force_treeallocate()/forcetree.c/4820
  23 2980 0            Father    221.9056   26804.3159  force_treeallocate()/forcetree.c/4827
  23 2981 0           Ngblist     63.4297   26867.7456  subfind_density()/subfind_density.c/78
  23 2982 0         Dist2list    126.8594   26994.6050  subfind_density()/subfind_density.c/79
  23 2983 0    DataIndexTable     21.8182   27016.4232  subfind_density()/subfind_density.c/87
  23 2984 0      DataNodeList     58.1818   27074.6050  subfind_density()/subfind_density.c/89
  23 2985 0              Left    126.8594   27201.4644  subfind_density()/subfind_density.c/91
  23 2986 0             Right    126.8594   27328.3239  subfind_density()/subfind_density.c/92
  23 2987 0              Todo     15.8574   27344.1813  subfind_density()/subfind_density.c/93
  23 2988 0             DM_Vx    126.8594   27471.0407  subfind_density()/subfind_density.c/95
  23 2989 0             DM_Vy    126.8594   27597.9002  subfind_density()/subfind_density.c/96
  23 2990 0             DM_Vz    126.8594   27724.7596  subfind_density()/subfind_density.c/97
  23 2991 0    	  DensDataGet    129.9428   27854.7024  subfind_density()/subfind_density.c/161
  23 2992 0  	  DensDataResult     97.4571   27952.1595  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=23: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=47.8405 MB).
task 23: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 23
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 101 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 101 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 101 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 101 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 101 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 101 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 101 2973 0         FOF_PList    380.4932   18861.2950  fof_fof()/fof.c/249
 101 2974 0         FOF_GList    634.1553   19495.4503  fof_fof()/fof.c/337
 101 2975 0             Group     11.6989   19507.1492  fof_fof()/fof.c/380
 101 2976 0   DomainNodeIndex      0.3461   19507.4953  force_treeallocate()/forcetree.c/4800
 101 2977 0        Nodes_base   3737.5076   23245.0030  force_treeallocate()/forcetree.c/4803
 101 2978 0     Extnodes_base   3114.5897   26359.5927  force_treeallocate()/forcetree.c/4811
 101 2979 0          Nextnode    222.3012   26581.8938  force_treeallocate()/forcetree.c/4820
 101 2980 0            Father    221.9056   26803.7994  force_treeallocate()/forcetree.c/4827
 101 2981 0           Ngblist     63.4155   26867.2149  subfind_density()/subfind_density.c/78
  26 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  26 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  26 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  26 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  26 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  26 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  26 2973 0         FOF_PList    380.4849   18861.2867  fof_fof()/fof.c/249
  26 2974 0         FOF_GList    634.1415   19495.4283  fof_fof()/fof.c/337
  26 2975 0             Group     10.3988   19505.8271  fof_fof()/fof.c/380
  26 2976 0   DomainNodeIndex      0.3461   19506.1732  force_treeallocate()/forcetree.c/4800
 101 2982 0         Dist2list    126.8311   26994.0460  subfind_density()/subfind_density.c/79
 101 2983 0    DataIndexTable     21.8182   27015.8642  subfind_density()/subfind_density.c/87
 101 2984 0      DataNodeList     58.1818   27074.0460  subfind_density()/subfind_density.c/89
 101 2985 0              Left    126.8311   27200.8770  subfind_density()/subfind_density.c/91
 101 2986 0             Right    126.8311   27327.7081  subfind_density()/subfind_density.c/92
  26 2977 0        Nodes_base   3737.5076   23243.6808  force_treeallocate()/forcetree.c/4803
  26 2978 0     Extnodes_base   3114.5897   26358.2705  force_treeallocate()/forcetree.c/4811
 101 2987 0              Todo     15.8539   27343.5620  subfind_density()/subfind_density.c/93
 101 2988 0             DM_Vx    126.8311   27470.3930  subfind_density()/subfind_density.c/95
 101 2989 0             DM_Vy    126.8311   27597.2241  subfind_density()/subfind_density.c/96
  26 2979 0          Nextnode    222.3012   26580.5717  force_treeallocate()/forcetree.c/4820
  26 2980 0            Father    221.9056   26802.4773  force_treeallocate()/forcetree.c/4827
  26 2981 0           Ngblist     63.4142   26865.8914  subfind_density()/subfind_density.c/78
  26 2982 0         Dist2list    126.8283   26992.7197  subfind_density()/subfind_density.c/79
 101 2990 0             DM_Vz    126.8311   27724.0552  subfind_density()/subfind_density.c/97
 101 2991 0    	  DensDataGet    115.7166   27839.7717  subfind_density()/subfind_density.c/161
 101 2992 0  	  DensDataResult     86.7874   27926.5591  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  26 2983 0    DataIndexTable     21.8182   27014.5379  subfind_density()/subfind_density.c/87
  26 2984 0      DataNodeList     58.1818   27072.7197  subfind_density()/subfind_density.c/89
  26 2985 0              Left    126.8283   27199.5480  subfind_density()/subfind_density.c/91

Task=101: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.4409 MB).
task 101: endrun called with an error level of 812


  26 2986 0             Right    126.8283   27326.3763  subfind_density()/subfind_density.c/92
  26 2987 0              Todo     15.8535   27342.2298  subfind_density()/subfind_density.c/93
  26 2988 0             DM_Vx    126.8283   27469.0582  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 101
  26 2989 0             DM_Vy    126.8283   27595.8865  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  26 2990 0             DM_Vz    126.8283   27722.7148  subfind_density()/subfind_density.c/97
 100 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 100 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 100 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 100 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 100 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 100 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  26 2991 0    	  DensDataGet    119.6188   27842.3336  subfind_density()/subfind_density.c/161
 100 2973 0         FOF_PList    380.3746   18861.1764  fof_fof()/fof.c/249
 100 2974 0         FOF_GList    633.9577   19495.1342  fof_fof()/fof.c/337
 100 2975 0             Group     10.3220   19505.4562  fof_fof()/fof.c/380
  26 2992 0  	  DensDataResult     89.7141   27932.0477  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 100 2976 0   DomainNodeIndex      0.3461   19505.8023  force_treeallocate()/forcetree.c/4800
 100 2977 0        Nodes_base   3737.5076   23243.3100  force_treeallocate()/forcetree.c/4803
 100 2978 0     Extnodes_base   3114.5897   26357.8997  force_treeallocate()/forcetree.c/4811

Task=26: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.9523 MB).
task 26: endrun called with an error level of 812


 100 2979 0          Nextnode    222.3012   26580.2008  force_treeallocate()/forcetree.c/4820
 100 2980 0            Father    221.9056   26802.1064  force_treeallocate()/forcetree.c/4827
 100 2981 0           Ngblist     63.3958   26865.5022  subfind_density()/subfind_density.c/78
 100 2982 0         Dist2list    126.7915   26992.2937  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 26
 100 2983 0    DataIndexTable     21.8182   27014.1119  subfind_density()/subfind_density.c/87
 100 2984 0      DataNodeList     58.1818   27072.2937  subfind_density()/subfind_density.c/89
 100 2985 0              Left    126.7915   27199.0853  subfind_density()/subfind_density.c/91
 100 2986 0             Right    126.7915   27325.8768  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 100 2987 0              Todo     15.8489   27341.7257  subfind_density()/subfind_density.c/93
 100 2988 0             DM_Vx    126.7915   27468.5173  subfind_density()/subfind_density.c/95
  27 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  27 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  27 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  27 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  27 2973 0         FOF_PList    380.4686   18861.2704  fof_fof()/fof.c/249
  27 2974 0         FOF_GList    634.1143   19495.3847  fof_fof()/fof.c/337
  27 2975 0             Group     11.5536   19506.9383  fof_fof()/fof.c/380
  27 2976 0   DomainNodeIndex      0.3461   19507.2845  force_treeallocate()/forcetree.c/4800
  27 2977 0        Nodes_base   3737.5076   23244.7921  force_treeallocate()/forcetree.c/4803
  27 2978 0     Extnodes_base   3114.5897   26359.3818  force_treeallocate()/forcetree.c/4811
 100 2989 0             DM_Vy    126.7915   27595.3088  subfind_density()/subfind_density.c/96
  27 2979 0          Nextnode    222.3012   26581.6829  force_treeallocate()/forcetree.c/4820
  27 2980 0            Father    221.9056   26803.5885  force_treeallocate()/forcetree.c/4827
  27 2981 0           Ngblist     63.4114   26867.0000  subfind_density()/subfind_density.c/78
 100 2990 0             DM_Vz    126.7915   27722.1004  subfind_density()/subfind_density.c/97
  27 2982 0         Dist2list    126.8229   26993.8228  subfind_density()/subfind_density.c/79
  27 2983 0    DataIndexTable     21.8182   27015.6410  subfind_density()/subfind_density.c/87
  27 2984 0      DataNodeList     58.1818   27073.8228  subfind_density()/subfind_density.c/89
  27 2985 0              Left    126.8229   27200.6456  subfind_density()/subfind_density.c/91
 100 2991 0    	  DensDataGet    115.4998   27837.6002  subfind_density()/subfind_density.c/161
  27 2986 0             Right    126.8229   27327.4685  subfind_density()/subfind_density.c/92
  27 2987 0              Todo     15.8529   27343.3214  subfind_density()/subfind_density.c/93
  27 2988 0             DM_Vx    126.8229   27470.1442  subfind_density()/subfind_density.c/95
  27 2989 0             DM_Vy    126.8229   27596.9671  subfind_density()/subfind_density.c/96
 100 2992 0  	  DensDataResult     86.6248   27924.2250  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  27 2990 0             DM_Vz    126.8229   27723.7899  subfind_density()/subfind_density.c/97
  27 2991 0    	  DensDataGet    112.2949   27836.0849  subfind_density()/subfind_density.c/161
  27 2992 0  	  DensDataResult     84.2212   27920.3061  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------


Task=27: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.6939 MB).
task 27: endrun called with an error level of 812



Task=100: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.775 MB).
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 27
task 100: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 100
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  97 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  97 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  97 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  97 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  97 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  97 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  97 2973 0         FOF_PList    380.4382   18861.2400  fof_fof()/fof.c/249
  97 2974 0         FOF_GList    634.0636   19495.3036  fof_fof()/fof.c/337
  97 2975 0             Group     10.9124   19506.2160  fof_fof()/fof.c/380
  97 2976 0   DomainNodeIndex      0.3461   19506.5621  force_treeallocate()/forcetree.c/4800
  97 2977 0        Nodes_base   3737.5076   23244.0697  force_treeallocate()/forcetree.c/4803
  97 2978 0     Extnodes_base   3114.5897   26358.6594  force_treeallocate()/forcetree.c/4811
  97 2979 0          Nextnode    222.3012   26580.9606  force_treeallocate()/forcetree.c/4820
  97 2980 0            Father    221.9056   26802.8662  force_treeallocate()/forcetree.c/4827
  97 2981 0           Ngblist     63.4064   26866.2725  subfind_density()/subfind_density.c/78
  97 2982 0         Dist2list    126.8127   26993.0853  subfind_density()/subfind_density.c/79
  97 2983 0    DataIndexTable     21.8182   27014.9034  subfind_density()/subfind_density.c/87
  97 2984 0      DataNodeList     58.1818   27073.0852  subfind_density()/subfind_density.c/89
  97 2985 0              Left    126.8127   27199.8979  subfind_density()/subfind_density.c/91
  97 2986 0             Right    126.8127   27326.7107  subfind_density()/subfind_density.c/92
  97 2987 0              Todo     15.8516   27342.5623  subfind_density()/subfind_density.c/93
  97 2988 0             DM_Vx    126.8127   27469.3750  subfind_density()/subfind_density.c/95
  97 2989 0             DM_Vy    126.8127   27596.1877  subfind_density()/subfind_density.c/96
  97 2990 0             DM_Vz    126.8127   27723.0004  subfind_density()/subfind_density.c/97
  97 2991 0    	  DensDataGet    119.9453   27842.9457  subfind_density()/subfind_density.c/161
  97 2992 0  	  DensDataResult     89.9590   27932.9047  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=97: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.0953 MB).
task 97: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  96 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  96 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  96 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  96 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  96 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  96 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  96 2973 0         FOF_PList    380.5372   18861.3390  fof_fof()/fof.c/249
  96 2974 0         FOF_GList    634.2287   19495.5677  fof_fof()/fof.c/337
  96 2975 0             Group     11.3503   19506.9180  fof_fof()/fof.c/380
  96 2976 0   DomainNodeIndex      0.3461   19507.2641  force_treeallocate()/forcetree.c/4800
  96 2977 0        Nodes_base   3737.5076   23244.7718  force_treeallocate()/forcetree.c/4803
  96 2978 0     Extnodes_base   3114.5897   26359.3615  force_treeallocate()/forcetree.c/4811
  96 2979 0          Nextnode    222.3012   26581.6626  force_treeallocate()/forcetree.c/4820
  96 2980 0            Father    221.9056   26803.5682  force_treeallocate()/forcetree.c/4827
  96 2981 0           Ngblist     63.4229   26866.9911  subfind_density()/subfind_density.c/78
  96 2982 0         Dist2list    126.8457   26993.8368  subfind_density()/subfind_density.c/79
  96 2983 0    DataIndexTable     21.8182   27015.6550  subfind_density()/subfind_density.c/87
  96 2984 0      DataNodeList     58.1818   27073.8368  subfind_density()/subfind_density.c/89
  96 2985 0              Left    126.8457   27200.6825  subfind_density()/subfind_density.c/91
  96 2986 0             Right    126.8457   27327.5283  subfind_density()/subfind_density.c/92
  96 2987 0              Todo     15.8557   27343.3840  subfind_density()/subfind_density.c/93
  96 2988 0             DM_Vx    126.8457   27470.2297  subfind_density()/subfind_density.c/95
  96 2989 0             DM_Vy    126.8457   27597.0755  subfind_density()/subfind_density.c/96
  96 2990 0             DM_Vz    126.8457   27723.9212  subfind_density()/subfind_density.c/97
  96 2991 0    	  DensDataGet    118.3056   27842.2268  subfind_density()/subfind_density.c/161
  96 2992 0  	  DensDataResult     88.7292   27930.9560  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=96: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.044 MB).
task 96: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 96
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 96:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
task 96: endrun called with an error level of 124


    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  25 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  25 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  25 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  25 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  25 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  25 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  25 2973 0         FOF_PList    380.5646   18861.3664  fof_fof()/fof.c/249
  25 2974 0         FOF_GList    634.2743   19495.6407  fof_fof()/fof.c/337
  25 2975 0             Group     13.1163   19508.7569  fof_fof()/fof.c/380
  25 2976 0   DomainNodeIndex      0.3461   19509.1031  force_treeallocate()/forcetree.c/4800
  25 2977 0        Nodes_base   3737.5076   23246.6107  force_treeallocate()/forcetree.c/4803
  25 2978 0     Extnodes_base   3114.5897   26361.2004  force_treeallocate()/forcetree.c/4811
  25 2979 0          Nextnode    222.3012   26583.5015  force_treeallocate()/forcetree.c/4820
  25 2980 0            Father    221.9056   26805.4071  force_treeallocate()/forcetree.c/4827
 102 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 102 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 102 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 102 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 102 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 102 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 102 2973 0         FOF_PList    380.4185   18861.2203  fof_fof()/fof.c/249
 102 2974 0         FOF_GList    634.0308   19495.2511  fof_fof()/fof.c/337
 102 2975 0             Group     11.2486   19506.4997  fof_fof()/fof.c/380
  25 2981 0           Ngblist     63.4274   26868.8346  subfind_density()/subfind_density.c/78
  25 2982 0         Dist2list    126.8549   26995.6894  subfind_density()/subfind_density.c/79
  25 2983 0    DataIndexTable     21.8182   27017.5076  subfind_density()/subfind_density.c/87
  25 2984 0      DataNodeList     58.1818   27075.6894  subfind_density()/subfind_density.c/89
 102 2976 0   DomainNodeIndex      0.3461   19506.8458  force_treeallocate()/forcetree.c/4800
 102 2977 0        Nodes_base   3737.5076   23244.3534  force_treeallocate()/forcetree.c/4803
 102 2978 0     Extnodes_base   3114.5897   26358.9431  force_treeallocate()/forcetree.c/4811
  25 2985 0              Left    126.8549   27202.5442  subfind_density()/subfind_density.c/91
  25 2986 0             Right    126.8549   27329.3991  subfind_density()/subfind_density.c/92
 102 2979 0          Nextnode    222.3012   26581.2443  force_treeallocate()/forcetree.c/4820
 102 2980 0            Father    221.9056   26803.1499  force_treeallocate()/forcetree.c/4827
 102 2981 0           Ngblist     63.4031   26866.5529  subfind_density()/subfind_density.c/78
 102 2982 0         Dist2list    126.8062   26993.3591  subfind_density()/subfind_density.c/79
  25 2987 0              Todo     15.8569   27345.2560  subfind_density()/subfind_density.c/93
  25 2988 0             DM_Vx    126.8549   27472.1108  subfind_density()/subfind_density.c/95
  25 2989 0             DM_Vy    126.8549   27598.9657  subfind_density()/subfind_density.c/96
 102 2983 0    DataIndexTable     21.8182   27015.1773  subfind_density()/subfind_density.c/87
 102 2984 0      DataNodeList     58.1818   27073.3591  subfind_density()/subfind_density.c/89
  25 2990 0             DM_Vz    126.8549   27725.8205  subfind_density()/subfind_density.c/97
  25 2991 0    	  DensDataGet    114.2115   27840.0320  subfind_density()/subfind_density.c/161
  25 2992 0  	  DensDataResult     85.6586   27925.6906  subfind_density()/subfind_density.c/201
 102 2985 0              Left    126.8062   27200.1652  subfind_density()/subfind_density.c/91
 102 2986 0             Right    126.8062   27326.9714  subfind_density()/subfind_density.c/92
 102 2987 0              Todo     15.8508   27342.8222  subfind_density()/subfind_density.c/93
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 102 2988 0             DM_Vx    126.8062   27469.6283  subfind_density()/subfind_density.c/95

Task=25: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.3094 MB).
task 25: endrun called with an error level of 812


   5 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   5 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   5 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   5 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   5 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   5 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
   5 2973 0         FOF_PList    380.5112   18861.3130  fof_fof()/fof.c/249
   5 2974 0         FOF_GList    634.1854   19495.4984  fof_fof()/fof.c/337
   5 2975 0             Group     11.7155   19507.2140  fof_fof()/fof.c/380
   5 2976 0   DomainNodeIndex      0.3461   19507.5601  force_treeallocate()/forcetree.c/4800
   5 2977 0        Nodes_base   3737.5076   23245.0677  force_treeallocate()/forcetree.c/4803
   5 2978 0     Extnodes_base   3114.5897   26359.6574  force_treeallocate()/forcetree.c/4811
 102 2989 0             DM_Vy    126.8062   27596.4345  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 25
   5 2979 0          Nextnode    222.3012   26581.9586  force_treeallocate()/forcetree.c/4820
   5 2980 0            Father    221.9056   26803.8642  force_treeallocate()/forcetree.c/4827
   5 2981 0           Ngblist     63.4185   26867.2827  subfind_density()/subfind_density.c/78
   5 2982 0         Dist2list    126.8371   26994.1198  subfind_density()/subfind_density.c/79
 102 2990 0             DM_Vz    126.8062   27723.2407  subfind_density()/subfind_density.c/97
   5 2983 0    DataIndexTable     21.8182   27015.9380  subfind_density()/subfind_density.c/87
   5 2984 0      DataNodeList     58.1818   27074.1198  subfind_density()/subfind_density.c/89
   5 2985 0              Left    126.8371   27200.9568  subfind_density()/subfind_density.c/91
 102 2991 0    	  DensDataGet    149.8059   27873.0466  subfind_density()/subfind_density.c/161
   5 2986 0             Right    126.8371   27327.7939  subfind_density()/subfind_density.c/92
   5 2987 0              Todo     15.8546   27343.6486  subfind_density()/subfind_density.c/93
   5 2988 0             DM_Vx    126.8371   27470.4856  subfind_density()/subfind_density.c/95
 102 2992 0  	  DensDataResult    112.3544   27985.4010  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   5 2989 0             DM_Vy    126.8371   27597.3227  subfind_density()/subfind_density.c/96
   5 2990 0             DM_Vz    126.8371   27724.1598  subfind_density()/subfind_density.c/97
   5 2991 0    	  DensDataGet    145.2390   27869.3988  subfind_density()/subfind_density.c/161

Task=102: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=14.599 MB).
task 102: endrun called with an error level of 812


   5 2992 0  	  DensDataResult    108.9292   27978.3280  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 102

Task=5: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=21.672 MB).
task 5: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 5
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  20 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  20 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  20 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  20 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  20 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  20 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 107 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 107 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 107 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 107 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 107 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 107 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 107 2973 0         FOF_PList    380.1159   18860.9177  fof_fof()/fof.c/249
 107 2974 0         FOF_GList    633.5266   19494.4443  fof_fof()/fof.c/337
 107 2975 0             Group     12.5902   19507.0345  fof_fof()/fof.c/380
 107 2976 0   DomainNodeIndex      0.3461   19507.3806  force_treeallocate()/forcetree.c/4800
 107 2977 0        Nodes_base   3737.5076   23244.8883  force_treeallocate()/forcetree.c/4803
 107 2978 0     Extnodes_base   3114.5897   26359.4780  force_treeallocate()/forcetree.c/4811
 107 2979 0          Nextnode    222.3012   26581.7791  force_treeallocate()/forcetree.c/4820
 107 2980 0            Father    221.9056   26803.6847  force_treeallocate()/forcetree.c/4827
 107 2981 0           Ngblist     63.3527   26867.0374  subfind_density()/subfind_density.c/78
  20 2973 0         FOF_PList    380.5772   18861.3790  fof_fof()/fof.c/249
  20 2974 0         FOF_GList    634.2953   19495.6742  fof_fof()/fof.c/337
  20 2975 0             Group     12.3599   19508.0341  fof_fof()/fof.c/380
 107 2982 0         Dist2list    126.7053   26993.7427  subfind_density()/subfind_density.c/79
 107 2983 0    DataIndexTable     21.8182   27015.5609  subfind_density()/subfind_density.c/87
 107 2984 0      DataNodeList     58.1818   27073.7427  subfind_density()/subfind_density.c/89
 107 2985 0              Left    126.7053   27200.4480  subfind_density()/subfind_density.c/91
  20 2976 0   DomainNodeIndex      0.3461   19508.3802  force_treeallocate()/forcetree.c/4800
  20 2977 0        Nodes_base   3737.5076   23245.8878  force_treeallocate()/forcetree.c/4803
  20 2978 0     Extnodes_base   3114.5897   26360.4775  force_treeallocate()/forcetree.c/4811
 107 2986 0             Right    126.7053   27327.1533  subfind_density()/subfind_density.c/92
 107 2987 0              Todo     15.8382   27342.9915  subfind_density()/subfind_density.c/93
 107 2988 0             DM_Vx    126.7053   27469.6968  subfind_density()/subfind_density.c/95
 107 2989 0             DM_Vy    126.7053   27596.4021  subfind_density()/subfind_density.c/96
  20 2979 0          Nextnode    222.3012   26582.7787  force_treeallocate()/forcetree.c/4820
  20 2980 0            Father    221.9056   26804.6843  force_treeallocate()/forcetree.c/4827
  20 2981 0           Ngblist     63.4295   26868.1138  subfind_density()/subfind_density.c/78
 107 2990 0             DM_Vz    126.7053   27723.1074  subfind_density()/subfind_density.c/97
 107 2991 0    	  DensDataGet    128.6752   27851.7826  subfind_density()/subfind_density.c/161
 107 2992 0  	  DensDataResult     96.5064   27948.2891  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  20 2982 0         Dist2list    126.8591   26994.9729  subfind_density()/subfind_density.c/79
  20 2983 0    DataIndexTable     21.8182   27016.7910  subfind_density()/subfind_density.c/87


Task=107: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.7109 MB).
task 107: endrun called with an error level of 812


  20 2984 0      DataNodeList     58.1818   27074.9728  subfind_density()/subfind_density.c/89
  20 2985 0              Left    126.8591   27201.8319  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 107
  20 2986 0             Right    126.8591   27328.6909  subfind_density()/subfind_density.c/92
  20 2987 0              Todo     15.8574   27344.5483  subfind_density()/subfind_density.c/93
  20 2988 0             DM_Vx    126.8591   27471.4074  subfind_density()/subfind_density.c/95
  20 2989 0             DM_Vy    126.8591   27598.2664  subfind_density()/subfind_density.c/96
  20 2990 0             DM_Vz    126.8591   27725.1255  subfind_density()/subfind_density.c/97
  20 2991 0    	  DensDataGet    111.2695   27836.3950  subfind_density()/subfind_density.c/161
  20 2992 0  	  DensDataResult     83.4521   27919.8471  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=20: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.1529 MB).
task 20: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 20
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  21 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  21 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  21 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  21 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  21 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  21 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  21 2973 0         FOF_PList    380.5812   18861.3830  fof_fof()/fof.c/249
  21 2974 0         FOF_GList    634.3020   19495.6850  fof_fof()/fof.c/337
  21 2975 0             Group     11.7560   19507.4409  fof_fof()/fof.c/380
  21 2976 0   DomainNodeIndex      0.3461   19507.7871  force_treeallocate()/forcetree.c/4800
  21 2977 0        Nodes_base   3737.5076   23245.2947  force_treeallocate()/forcetree.c/4803
 106 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 106 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 106 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 106 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 106 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 106 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 106 2973 0         FOF_PList    380.2913   18861.0931  fof_fof()/fof.c/249
  21 2978 0     Extnodes_base   3114.5897   26359.8844  force_treeallocate()/forcetree.c/4811
  21 2979 0          Nextnode    222.3012   26582.1856  force_treeallocate()/forcetree.c/4820
  21 2980 0            Father    221.9056   26804.0911  force_treeallocate()/forcetree.c/4827
  21 2981 0           Ngblist     63.4302   26867.5213  subfind_density()/subfind_density.c/78
  21 2982 0         Dist2list    126.8604   26994.3817  subfind_density()/subfind_density.c/79
 106 2974 0         FOF_GList    633.8188   19494.9119  fof_fof()/fof.c/337
 106 2975 0             Group     12.5093   19507.4212  fof_fof()/fof.c/380
 106 2976 0   DomainNodeIndex      0.3461   19507.7673  force_treeallocate()/forcetree.c/4800
 106 2977 0        Nodes_base   3737.5076   23245.2749  force_treeallocate()/forcetree.c/4803
  21 2983 0    DataIndexTable     21.8182   27016.1999  subfind_density()/subfind_density.c/87
  21 2984 0      DataNodeList     58.1818   27074.3817  subfind_density()/subfind_density.c/89
  21 2985 0              Left    126.8604   27201.2421  subfind_density()/subfind_density.c/91
  21 2986 0             Right    126.8604   27328.1025  subfind_density()/subfind_density.c/92
 106 2978 0     Extnodes_base   3114.5897   26359.8646  force_treeallocate()/forcetree.c/4811
 106 2979 0          Nextnode    222.3012   26582.1658  force_treeallocate()/forcetree.c/4820
  21 2987 0              Todo     15.8576   27343.9601  subfind_density()/subfind_density.c/93
  21 2988 0             DM_Vx    126.8604   27470.8205  subfind_density()/subfind_density.c/95
  21 2989 0             DM_Vy    126.8604   27597.6809  subfind_density()/subfind_density.c/96
 106 2980 0            Father    221.9056   26804.0714  force_treeallocate()/forcetree.c/4827
 106 2981 0           Ngblist     63.3819   26867.4533  subfind_density()/subfind_density.c/78
 106 2982 0         Dist2list    126.7638   26994.2170  subfind_density()/subfind_density.c/79
  21 2990 0             DM_Vz    126.8604   27724.5413  subfind_density()/subfind_density.c/97
  21 2991 0    	  DensDataGet    119.3992   27843.9404  subfind_density()/subfind_density.c/161
  21 2992 0  	  DensDataResult     89.5494   27933.4898  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 106 2983 0    DataIndexTable     21.8182   27016.0352  subfind_density()/subfind_density.c/87
 106 2984 0      DataNodeList     58.1818   27074.2170  subfind_density()/subfind_density.c/89
 106 2985 0              Left    126.7638   27200.9808  subfind_density()/subfind_density.c/91
 106 2986 0             Right    126.7638   27327.7445  subfind_density()/subfind_density.c/92

Task=21: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.5102 MB).
task 21: endrun called with an error level of 812


 106 2987 0              Todo     15.8455   27343.5900  subfind_density()/subfind_density.c/93
 106 2988 0             DM_Vx    126.7638   27470.3538  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 21
 106 2989 0             DM_Vy    126.7638   27597.1175  subfind_density()/subfind_density.c/96
 106 2990 0             DM_Vz    126.7638   27723.8813  subfind_density()/subfind_density.c/97
 106 2991 0    	  DensDataGet    145.8729   27869.7541  subfind_density()/subfind_density.c/161
 106 2992 0  	  DensDataResult    109.4046   27979.1588  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=106: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=20.8412 MB).
task 106: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 106
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  29 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  29 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  29 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  29 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  29 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  29 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  29 2973 0         FOF_PList    380.2968   18861.0986  fof_fof()/fof.c/249
  29 2974 0         FOF_GList    633.8280   19494.9267  fof_fof()/fof.c/337
  29 2975 0             Group     10.8232   19505.7498  fof_fof()/fof.c/380
  29 2976 0   DomainNodeIndex      0.3461   19506.0960  force_treeallocate()/forcetree.c/4800
  29 2977 0        Nodes_base   3737.5076   23243.6036  force_treeallocate()/forcetree.c/4803
  29 2978 0     Extnodes_base   3114.5897   26358.1933  force_treeallocate()/forcetree.c/4811
  29 2979 0          Nextnode    222.3012   26580.4945  force_treeallocate()/forcetree.c/4820
  29 2980 0            Father    221.9056   26802.4000  force_treeallocate()/forcetree.c/4827
  29 2981 0           Ngblist     63.3828   26865.7829  subfind_density()/subfind_density.c/78
  98 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  98 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  98 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  98 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  98 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  98 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  98 2973 0         FOF_PList    380.5745   18861.3763  fof_fof()/fof.c/249
  98 2974 0         FOF_GList    634.2909   19495.6672  fof_fof()/fof.c/337
  98 2975 0             Group     12.5446   19508.2118  fof_fof()/fof.c/380
  98 2976 0   DomainNodeIndex      0.3461   19508.5579  force_treeallocate()/forcetree.c/4800
  98 2977 0        Nodes_base   3737.5076   23246.0655  force_treeallocate()/forcetree.c/4803
  29 2982 0         Dist2list    126.7656   26992.5485  subfind_density()/subfind_density.c/79
  29 2983 0    DataIndexTable     21.8182   27014.3666  subfind_density()/subfind_density.c/87
  29 2984 0      DataNodeList     58.1818   27072.5484  subfind_density()/subfind_density.c/89
  29 2985 0              Left    126.7656   27199.3140  subfind_density()/subfind_density.c/91
  98 2978 0     Extnodes_base   3114.5897   26360.6552  force_treeallocate()/forcetree.c/4811
  98 2979 0          Nextnode    222.3012   26582.9564  force_treeallocate()/forcetree.c/4820
  29 2986 0             Right    126.7656   27326.0797  subfind_density()/subfind_density.c/92
  29 2987 0              Todo     15.8457   27341.9254  subfind_density()/subfind_density.c/93
  98 2980 0            Father    221.9056   26804.8620  force_treeallocate()/forcetree.c/4827
  98 2981 0           Ngblist     63.4291   26868.2911  subfind_density()/subfind_density.c/78
  98 2982 0         Dist2list    126.8582   26995.1492  subfind_density()/subfind_density.c/79
  29 2988 0             DM_Vx    126.7656   27468.6910  subfind_density()/subfind_density.c/95
  29 2989 0             DM_Vy    126.7656   27595.4566  subfind_density()/subfind_density.c/96
  98 2983 0    DataIndexTable     21.8182   27016.9674  subfind_density()/subfind_density.c/87
  98 2984 0      DataNodeList     58.1818   27075.1492  subfind_density()/subfind_density.c/89
  98 2985 0              Left    126.8582   27202.0074  subfind_density()/subfind_density.c/91
  98 2986 0             Right    126.8582   27328.8656  subfind_density()/subfind_density.c/92
  98 2987 0              Todo     15.8573   27344.7228  subfind_density()/subfind_density.c/93
  98 2988 0             DM_Vx    126.8582   27471.5810  subfind_density()/subfind_density.c/95
  98 2989 0             DM_Vy    126.8582   27598.4392  subfind_density()/subfind_density.c/96
  98 2990 0             DM_Vz    126.8582   27725.2974  subfind_density()/subfind_density.c/97
  29 2990 0             DM_Vz    126.7656   27722.2222  subfind_density()/subfind_density.c/97
  29 2991 0    	  DensDataGet    133.6573   27855.8795  subfind_density()/subfind_density.c/161
  29 2992 0  	  DensDataResult    100.2430   27956.1224  subfind_density()/subfind_density.c/201
  98 2991 0    	  DensDataGet    125.8102   27851.1076  subfind_density()/subfind_density.c/161
  98 2992 0  	  DensDataResult     94.3577   27945.4653  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------

Task=29: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=43.8776 MB).

Task=98: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.5347 MB).
task 98: endrun called with an error level of 812


task 29: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 98
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 29
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   4 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   4 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   4 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   4 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   4 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   4 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 123 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 123 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 123 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 123 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 123 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 123 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 123 2973 0         FOF_PList    380.5851   18861.3869  fof_fof()/fof.c/249
 123 2974 0         FOF_GList    634.3085   19495.6955  fof_fof()/fof.c/337
 123 2975 0             Group     12.1938   19507.8893  fof_fof()/fof.c/380
 123 2976 0   DomainNodeIndex      0.3461   19508.2354  force_treeallocate()/forcetree.c/4800
 123 2977 0        Nodes_base   3737.5076   23245.7431  force_treeallocate()/forcetree.c/4803
 123 2978 0     Extnodes_base   3114.5897   26360.3328  force_treeallocate()/forcetree.c/4811
 123 2979 0          Nextnode    222.3012   26582.6339  force_treeallocate()/forcetree.c/4820
   4 2973 0         FOF_PList    379.9014   18860.7032  fof_fof()/fof.c/249
   4 2974 0         FOF_GList    633.1689   19493.8721  fof_fof()/fof.c/337
   4 2975 0             Group     10.7921   19504.6642  fof_fof()/fof.c/380
 123 2980 0            Father    221.9056   26804.5395  force_treeallocate()/forcetree.c/4827
 123 2981 0           Ngblist     63.4309   26867.9704  subfind_density()/subfind_density.c/78
 123 2982 0         Dist2list    126.8617   26994.8321  subfind_density()/subfind_density.c/79
 123 2983 0    DataIndexTable     21.8182   27016.6503  subfind_density()/subfind_density.c/87
 123 2984 0      DataNodeList     58.1818   27074.8320  subfind_density()/subfind_density.c/89
   4 2976 0   DomainNodeIndex      0.3461   19505.0103  force_treeallocate()/forcetree.c/4800
   4 2977 0        Nodes_base   3737.5076   23242.5179  force_treeallocate()/forcetree.c/4803
 123 2985 0              Left    126.8617   27201.6938  subfind_density()/subfind_density.c/91
 123 2986 0             Right    126.8617   27328.5555  subfind_density()/subfind_density.c/92
 123 2987 0              Todo     15.8577   27344.4132  subfind_density()/subfind_density.c/93
   4 2978 0     Extnodes_base   3114.5897   26357.1076  force_treeallocate()/forcetree.c/4811
   4 2979 0          Nextnode    222.3012   26579.4088  force_treeallocate()/forcetree.c/4820
   4 2980 0            Father    221.9056   26801.3144  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 123 2988 0             DM_Vx    126.8617   27471.2749  subfind_density()/subfind_density.c/95
 123 2989 0             DM_Vy    126.8617   27598.1366  subfind_density()/subfind_density.c/96
 123 2990 0             DM_Vz    126.8617   27724.9983  subfind_density()/subfind_density.c/97
   4 2981 0           Ngblist     63.3169   26864.6312  subfind_density()/subfind_density.c/78
   4 2982 0         Dist2list    126.6338   26991.2650  subfind_density()/subfind_density.c/79
   4 2983 0    DataIndexTable     21.8182   27013.0832  subfind_density()/subfind_density.c/87
 103 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 103 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 103 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 103 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 103 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 103 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 103 2973 0         FOF_PList    380.4576   18861.2594  fof_fof()/fof.c/249
 103 2974 0         FOF_GList    634.0960   19495.3554  fof_fof()/fof.c/337
 103 2975 0             Group     11.8919   19507.2473  fof_fof()/fof.c/380
 103 2976 0   DomainNodeIndex      0.3461   19507.5934  force_treeallocate()/forcetree.c/4800
 103 2977 0        Nodes_base   3737.5076   23245.1010  force_treeallocate()/forcetree.c/4803
 103 2978 0     Extnodes_base   3114.5897   26359.6907  force_treeallocate()/forcetree.c/4811
 103 2979 0          Nextnode    222.3012   26581.9919  force_treeallocate()/forcetree.c/4820
 103 2980 0            Father    221.9056   26803.8975  force_treeallocate()/forcetree.c/4827
  24 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  24 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  24 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  24 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  24 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  24 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
 123 2991 0    	  DensDataGet    117.0649   27842.0633  subfind_density()/subfind_density.c/161
 123 2992 0  	  DensDataResult     87.7987   27929.8620  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=123: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.138 MB).
task 123: endrun called with an error level of 812


   4 2984 0      DataNodeList     58.1818   27071.2650  subfind_density()/subfind_density.c/89
   4 2985 0              Left    126.6338   27197.8988  subfind_density()/subfind_density.c/91
   4 2986 0             Right    126.6338   27324.5326  subfind_density()/subfind_density.c/92
   4 2987 0              Todo     15.8292   27340.3618  subfind_density()/subfind_density.c/93
 103 2981 0           Ngblist     63.4096   26867.3071  subfind_density()/subfind_density.c/78
 103 2982 0         Dist2list    126.8192   26994.1263  subfind_density()/subfind_density.c/79
 103 2983 0    DataIndexTable     21.8182   27015.9445  subfind_density()/subfind_density.c/87
 103 2984 0      DataNodeList     58.1818   27074.1263  subfind_density()/subfind_density.c/89
  24 2973 0         FOF_PList    380.5691   18861.3709  fof_fof()/fof.c/249
  24 2974 0         FOF_GList    634.2818   19495.6527  fof_fof()/fof.c/337
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 123
   4 2988 0             DM_Vx    126.6338   27466.9956  subfind_density()/subfind_density.c/95
   4 2989 0             DM_Vy    126.6338   27593.6294  subfind_density()/subfind_density.c/96
 103 2985 0              Left    126.8192   27200.9454  subfind_density()/subfind_density.c/91
 103 2986 0             Right    126.8192   27327.7646  subfind_density()/subfind_density.c/92
 103 2987 0              Todo     15.8524   27343.6171  subfind_density()/subfind_density.c/93
 103 2988 0             DM_Vx    126.8192   27470.4362  subfind_density()/subfind_density.c/95
  24 2975 0             Group     11.3451   19506.9978  fof_fof()/fof.c/380
  24 2976 0   DomainNodeIndex      0.3461   19507.3440  force_treeallocate()/forcetree.c/4800
  24 2977 0        Nodes_base   3737.5076   23244.8516  force_treeallocate()/forcetree.c/4803
   4 2990 0             DM_Vz    126.6338   27720.2632  subfind_density()/subfind_density.c/97
 103 2989 0             DM_Vy    126.8192   27597.2554  subfind_density()/subfind_density.c/96
 103 2990 0             DM_Vz    126.8192   27724.0746  subfind_density()/subfind_density.c/97
 103 2991 0    	  DensDataGet    122.9412   27847.0159  subfind_density()/subfind_density.c/161
  24 2978 0     Extnodes_base   3114.5897   26359.4413  force_treeallocate()/forcetree.c/4811
  24 2979 0          Nextnode    222.3012   26581.7424  force_treeallocate()/forcetree.c/4820
   4 2991 0    	  DensDataGet    122.5681   27842.8313  subfind_density()/subfind_density.c/161
 103 2992 0  	  DensDataResult     92.2059   27939.2218  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=103: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.7782 MB).
task 103: endrun called with an error level of 812


  24 2980 0            Father    221.9056   26803.6480  force_treeallocate()/forcetree.c/4827
  24 2981 0           Ngblist     63.4282   26867.0762  subfind_density()/subfind_density.c/78
  24 2982 0         Dist2list    126.8564   26993.9326  subfind_density()/subfind_density.c/79
   4 2992 0  	  DensDataResult     91.9261   27934.7574  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 103
  24 2983 0    DataIndexTable     21.8182   27015.7508  subfind_density()/subfind_density.c/87
  24 2984 0      DataNodeList     58.1818   27073.9326  subfind_density()/subfind_density.c/89

Task=4: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.2426 MB).
task 4: endrun called with an error level of 812


  24 2985 0              Left    126.8564   27200.7889  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 4
  24 2986 0             Right    126.8564   27327.6453  subfind_density()/subfind_density.c/92
  24 2987 0              Todo     15.8570   27343.5023  subfind_density()/subfind_density.c/93
  24 2988 0             DM_Vx    126.8564   27470.3587  subfind_density()/subfind_density.c/95
  24 2989 0             DM_Vy    126.8564   27597.2151  subfind_density()/subfind_density.c/96
  24 2990 0             DM_Vz    126.8564   27724.0714  subfind_density()/subfind_density.c/97
  24 2991 0    	  DensDataGet    136.0702   27860.1416  subfind_density()/subfind_density.c/161
  24 2992 0  	  DensDataResult    102.0526   27962.1943  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=24: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=37.8057 MB).
task 24: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 24
task 24: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 24:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 24
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  28 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  28 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  28 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  28 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  99 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  99 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  99 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  99 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  99 2973 0         FOF_PList    380.3020   18861.1037  fof_fof()/fof.c/249
  99 2974 0         FOF_GList    633.8366   19494.9403  fof_fof()/fof.c/337
  99 2975 0             Group     11.6263   19506.5666  fof_fof()/fof.c/380
  99 2976 0   DomainNodeIndex      0.3461   19506.9128  force_treeallocate()/forcetree.c/4800
  99 2977 0        Nodes_base   3737.5076   23244.4204  force_treeallocate()/forcetree.c/4803
  99 2978 0     Extnodes_base   3114.5897   26359.0101  force_treeallocate()/forcetree.c/4811
  28 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  28 2972 0          TopNodes      5.1424   18480.8018  domain_allocate()/domain.c/570
  99 2979 0          Nextnode    222.3012   26581.3112  force_treeallocate()/forcetree.c/4820
  99 2980 0            Father    221.9056   26803.2168  force_treeallocate()/forcetree.c/4827
  99 2981 0           Ngblist     63.3837   26866.6005  subfind_density()/subfind_density.c/78
  99 2982 0         Dist2list    126.7673   26993.3678  subfind_density()/subfind_density.c/79
  28 2973 0         FOF_PList    380.5846   18861.3864  fof_fof()/fof.c/249
  28 2974 0         FOF_GList    634.3077   19495.6942  fof_fof()/fof.c/337
  99 2983 0    DataIndexTable     21.8182   27015.1860  subfind_density()/subfind_density.c/87
  99 2984 0      DataNodeList     58.1818   27073.3678  subfind_density()/subfind_density.c/89
  99 2985 0              Left    126.7673   27200.1351  subfind_density()/subfind_density.c/91
  99 2986 0             Right    126.7673   27326.9024  subfind_density()/subfind_density.c/92
  28 2975 0             Group     12.9202   19508.6143  fof_fof()/fof.c/380
  28 2976 0   DomainNodeIndex      0.3461   19508.9605  force_treeallocate()/forcetree.c/4800
  28 2977 0        Nodes_base   3737.5076   23246.4681  force_treeallocate()/forcetree.c/4803
  99 2987 0              Todo     15.8459   27342.7483  subfind_density()/subfind_density.c/93
  99 2988 0             DM_Vx    126.7673   27469.5156  subfind_density()/subfind_density.c/95
  99 2989 0             DM_Vy    126.7673   27596.2830  subfind_density()/subfind_density.c/96
  28 2978 0     Extnodes_base   3114.5897   26361.0578  force_treeallocate()/forcetree.c/4811
  28 2979 0          Nextnode    222.3012   26583.3590  force_treeallocate()/forcetree.c/4820
  99 2990 0             DM_Vz    126.7673   27723.0503  subfind_density()/subfind_density.c/97
  99 2991 0    	  DensDataGet    125.5814   27848.6317  subfind_density()/subfind_density.c/161
  99 2992 0  	  DensDataResult     94.1861   27942.8178  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  28 2980 0            Father    221.9056   26805.2645  force_treeallocate()/forcetree.c/4827
  28 2981 0           Ngblist     63.4308   26868.6953  subfind_density()/subfind_density.c/78

Task=99: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.1822 MB).
task 99: endrun called with an error level of 812


  28 2982 0         Dist2list    126.8615   26995.5569  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 99
  28 2983 0    DataIndexTable     21.8182   27017.3751  subfind_density()/subfind_density.c/87
  28 2984 0      DataNodeList     58.1818   27075.5568  subfind_density()/subfind_density.c/89
  28 2985 0              Left    126.8615   27202.4184  subfind_density()/subfind_density.c/91
  28 2986 0             Right    126.8615   27329.2799  subfind_density()/subfind_density.c/92
  28 2987 0              Todo     15.8577   27345.1376  subfind_density()/subfind_density.c/93
  28 2988 0             DM_Vx    126.8615   27471.9992  subfind_density()/subfind_density.c/95
  28 2989 0             DM_Vy    126.8615   27598.8607  subfind_density()/subfind_density.c/96
  28 2990 0             DM_Vz    126.8615   27725.7223  subfind_density()/subfind_density.c/97
  28 2991 0    	  DensDataGet    138.9610   27864.6833  subfind_density()/subfind_density.c/161
  28 2992 0  	  DensDataResult    104.2207   27968.9040  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=28: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=31.096 MB).
task 28: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 28
-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63110.18 Mb (on task=120), Smallest =   62840.46 Mb (on task=0), Average =   63039.95 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1691.31 Mb (on task=0), Smallest =    1421.88 Mb (on task=120), Average =    1492.13 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.11 Mb
AllocMem:	 Largest =    1691.31 Mb (on task=0), Smallest =    1421.88 Mb (on task=120), Average =    1492.13 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_025_z000p113/snap_025_z000p113.0.hdf5' on task=0 (contains 134499140 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  134171969  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      9893  (tot=     0000134208) masstab=0
Type 3 (bulge):   317278  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.898798

Setting next time for line of sight file to Time_next= 0.905318

Setting next time for snipshot file to Time_next= 0.906994

Setting next time for fof output to Time_next= 0.906994
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=7.22106 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=103353
After=104065
NTopleaves= 91057  NTopnodes=104065 (space for 465370)
gravity work-load balance=1.00035   memory-balance=1.00036   SPH work-load balance=0
iter=0 exchange of 2112803555 particles (ret=0)
domain decomposition done. (took 42.0704 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5392 MByte in top-level domain structure

Allocated 7296.37 MByte for BH-tree, and 0.347355 Mbyte for top-leaves.  (presently allocted 25777.5 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.4 Mbyte (on task=70), Smallest = 26119.9 Mbyte, Average = 26120.3 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  70 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  70 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  70 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  70 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  70 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  70 2972 0          TopNodes      5.1607   18480.8200  domain_allocate()/domain.c/570
  70 2973 0   DomainNodeIndex      0.3474   18481.1674  force_treeallocate()/forcetree.c/4800
  70 2974 0        Nodes_base   3737.5413   22218.7087  force_treeallocate()/forcetree.c/4803
  70 2975 0     Extnodes_base   3114.6178   25333.3265  force_treeallocate()/forcetree.c/4811
  70 2976 0          Nextnode    222.3026   25555.6291  force_treeallocate()/forcetree.c/4820
  70 2977 0            Father    221.9056   25777.5346  force_treeallocate()/forcetree.c/4827
  70 2978 0           Ngblist     63.4242   25840.9589  density()/density.c/690
  70 2979 0              Left    126.8484   25967.8073  density()/density.c/692
  70 2980 0             Right    126.8484   26094.6558  density()/density.c/693
  70 2981 0    DataIndexTable      7.0175   26101.6733  density()/density.c/738
  70 2982 0      DataNodeList     18.7134   26120.3867  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.101.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.155.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.5 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=94497
After=104713
NTopleaves= 91624  NTopnodes=104713 (space for 465370)
gravity work-load balance=1.00064   memory-balance=1.00065   SPH work-load balance=0
iter=0 exchange of 2110248087 particles (ret=0)
domain decomposition done. (took 33.1444 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5096 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.9 MB)
Tree construction.

Start linking particles (presently allocated=26536.9 MB)
links on local processor done (took 170.06 sec).
Marked=0233166990 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26860.4 MB) 
have done 0002946016 cross links (processed 0233166990, took 24.8358 sec)
have done 0001314498 cross links (processed 0194920676, took 24.918 sec)
have done 0000951011 cross links (processed 0168008693, took 21.9257 sec)
have done 0000637130 cross links (processed 0111459722, took 18.0339 sec)
have done 0000420389 cross links (processed 0075914667, took 14.9787 sec)
have done 0000258319 cross links (processed 0036274980, took 13.7958 sec)
have done 0000148228 cross links (processed 0019310894, took 4.68246 sec)
have done 0000075004 cross links (processed 0013189859, took 12.5574 sec)
have done 0000040818 cross links (processed 0007607903, took 10.271 sec)
have done 0000022719 cross links (processed 0000841187, took 0.675901 sec)
have done 0000012879 cross links (processed 0000388305, took 1.35854 sec)
have done 0000010122 cross links (processed 0000228774, took 0.534494 sec)
have done 0000005373 cross links (processed 0000166304, took 0.888978 sec)
have done 0000002397 cross links (processed 0000046011, took 0.193017 sec)
have done 0000000978 cross links (processed 0000014799, took 0.179853 sec)
have done 0000000331 cross links (processed 0000004599, took 0.185627 sec)
have done 0000000128 cross links (processed 0000001521, took 0.175539 sec)
have done 0000000045 cross links (processed 0000000570, took 0.180842 sec)
have done 0000000020 cross links (processed 0000000203, took 0.175227 sec)
have done 0000000005 cross links (processed 0000000076, took 0.174728 sec)
have done 0000000001 cross links (processed 0000000017, took 0.175468 sec)
have done 0000000000 cross links (processed 0000000005, took 0.174681 sec)
Local groups found.

group finding took = 321.677 sec
Start finding nearest dm-particle (presently allocated=26536.9 MB)
still finding nearest... (presently allocated=26803.9 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.486808 sec
compiling local group data and catalogue took = 9.15196 sec

Total number of groups with at least 32 particles: 1275466
Largest group has 139892247 particles.
Total number of particles in groups: 1305468897

group properties are now allocated.. (presently allocated=19505.6 MB)
computation of group properties took = 0.543564 sec
start global sorting of group catalogues number 25
LargestGroup = 109051.461977; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.213825 sec
Group catalogues globally sorted. took = 8.9323 sec
starting saving of group catalogue number 25
data//groups_025_z000p113
Group catalogues number 25 saved. took = 15.6496 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 11.1495 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 2.07156 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 8.34579 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  54 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  54 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  54 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  54 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  54 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  54 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  54 2973 0         FOF_PList    380.5199   18861.3720  fof_fof()/fof.c/249
  54 2974 0         FOF_GList    634.1998   19495.5718  fof_fof()/fof.c/337
  54 2975 0             Group     11.5277   19507.0995  fof_fof()/fof.c/380
  54 2976 0   DomainNodeIndex      0.3495   19507.4490  force_treeallocate()/forcetree.c/4800
  54 2977 0        Nodes_base   3737.6006   23245.0497  force_treeallocate()/forcetree.c/4803
  54 2978 0     Extnodes_base   3114.6672   26359.7169  force_treeallocate()/forcetree.c/4811
  54 2979 0          Nextnode    222.3050   26582.0219  force_treeallocate()/forcetree.c/4820
  54 2980 0            Father    221.9056   26803.9275  force_treeallocate()/forcetree.c/4827
  54 2981 0           Ngblist     63.4200   26867.3475  subfind_density()/subfind_density.c/78
  54 2982 0         Dist2list    126.8400   26994.1874  subfind_density()/subfind_density.c/79
  54 2983 0    DataIndexTable     21.8182   27016.0056  subfind_density()/subfind_density.c/87
  54 2984 0      DataNodeList     58.1818   27074.1874  subfind_density()/subfind_density.c/89
  54 2985 0              Left    126.8400   27201.0273  subfind_density()/subfind_density.c/91
  54 2986 0             Right    126.8400   27327.8673  subfind_density()/subfind_density.c/92
  54 2987 0              Todo     15.8550   27343.7223  subfind_density()/subfind_density.c/93
  54 2988 0             DM_Vx    126.8400   27470.5622  subfind_density()/subfind_density.c/95
  54 2989 0             DM_Vy    126.8400   27597.4022  subfind_density()/subfind_density.c/96
  54 2990 0             DM_Vz    126.8400   27724.2421  subfind_density()/subfind_density.c/97
  54 2991 0    	  DensDataGet    118.3799   27842.6221  subfind_density()/subfind_density.c/161
  54 2992 0  	  DensDataResult     88.7850   27931.4070  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=54: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.593 MB).
task 54: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 54
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  49 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  49 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  49 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  49 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  49 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  49 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  49 2973 0         FOF_PList    380.3653   18861.2174  fof_fof()/fof.c/249
  49 2974 0         FOF_GList    633.9421   19495.1595  fof_fof()/fof.c/337
  49 2975 0             Group     11.9656   19507.1251  fof_fof()/fof.c/380
  49 2976 0   DomainNodeIndex      0.3495   19507.4746  force_treeallocate()/forcetree.c/4800
  49 2977 0        Nodes_base   3737.6006   23245.0753  force_treeallocate()/forcetree.c/4803
  49 2978 0     Extnodes_base   3114.6672   26359.7425  force_treeallocate()/forcetree.c/4811
  49 2979 0          Nextnode    222.3050   26582.0475  force_treeallocate()/forcetree.c/4820
  49 2980 0            Father    221.9056   26803.9531  force_treeallocate()/forcetree.c/4827
  78 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  78 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  78 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  78 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  78 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  78 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  78 2973 0         FOF_PList    380.6316   18861.4838  fof_fof()/fof.c/249
  49 2981 0           Ngblist     63.3942   26867.3473  subfind_density()/subfind_density.c/78
  49 2982 0         Dist2list    126.7884   26994.1357  subfind_density()/subfind_density.c/79
  49 2983 0    DataIndexTable     21.8182   27015.9539  subfind_density()/subfind_density.c/87
  49 2984 0      DataNodeList     58.1818   27074.1357  subfind_density()/subfind_density.c/89
  49 2985 0              Left    126.7884   27200.9241  subfind_density()/subfind_density.c/91
  78 2974 0         FOF_GList    634.3860   19495.8698  fof_fof()/fof.c/337
  78 2975 0             Group     11.4955   19507.3653  fof_fof()/fof.c/380
  78 2976 0   DomainNodeIndex      0.3495   19507.7148  force_treeallocate()/forcetree.c/4800
  49 2986 0             Right    126.7884   27327.7126  subfind_density()/subfind_density.c/92
  49 2987 0              Todo     15.8486   27343.5611  subfind_density()/subfind_density.c/93
  49 2988 0             DM_Vx    126.7884   27470.3495  subfind_density()/subfind_density.c/95
  78 2977 0        Nodes_base   3737.6006   23245.3155  force_treeallocate()/forcetree.c/4803
  78 2978 0     Extnodes_base   3114.6672   26359.9827  force_treeallocate()/forcetree.c/4811
  78 2979 0          Nextnode    222.3050   26582.2877  force_treeallocate()/forcetree.c/4820
  49 2989 0             DM_Vy    126.7884   27597.1380  subfind_density()/subfind_density.c/96
  49 2990 0             DM_Vz    126.7884   27723.9264  subfind_density()/subfind_density.c/97
  49 2991 0    	  DensDataGet    159.1013   27883.0276  subfind_density()/subfind_density.c/161
----------------------------------------------------------------------------------------
  78 2980 0            Father    221.9056   26804.1933  force_treeallocate()/forcetree.c/4827
  78 2981 0           Ngblist     63.4386   26867.6319  subfind_density()/subfind_density.c/78

Task=49: Not enough memory in mymalloc_fullinfo() to allocate 119.326 MB for variable '	  DensDataResult' at subfind_density()/subfind_density.c/line 201 (FreeBytes=116.972 MB).
task 49: endrun called with an error level of 812


  78 2982 0         Dist2list    126.8772   26994.5091  subfind_density()/subfind_density.c/79
  78 2983 0    DataIndexTable     21.8182   27016.3273  subfind_density()/subfind_density.c/87
  78 2984 0      DataNodeList     58.1818   27074.5091  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 49
  78 2985 0              Left    126.8772   27201.3863  subfind_density()/subfind_density.c/91
  78 2986 0             Right    126.8772   27328.2635  subfind_density()/subfind_density.c/92
  78 2987 0              Todo     15.8596   27344.1231  subfind_density()/subfind_density.c/93
  78 2988 0             DM_Vx    126.8772   27471.0003  subfind_density()/subfind_density.c/95
  78 2989 0             DM_Vy    126.8772   27597.8775  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  78 2990 0             DM_Vz    126.8772   27724.7547  subfind_density()/subfind_density.c/97
  48 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  48 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  48 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  48 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  48 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  78 2991 0    	  DensDataGet    126.1086   27850.8633  subfind_density()/subfind_density.c/161
  48 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  48 2973 0         FOF_PList    380.0691   18860.9212  fof_fof()/fof.c/249
  48 2974 0         FOF_GList    633.4484   19494.3697  fof_fof()/fof.c/337
  48 2975 0             Group     11.2974   19505.6671  fof_fof()/fof.c/380
  78 2992 0  	  DensDataResult     94.5814   27945.4447  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  48 2976 0   DomainNodeIndex      0.3495   19506.0166  force_treeallocate()/forcetree.c/4800
  48 2977 0        Nodes_base   3737.6006   23243.6172  force_treeallocate()/forcetree.c/4803

  48 2978 0     Extnodes_base   3114.6672   26358.2844  force_treeallocate()/forcetree.c/4811
  48 2979 0          Nextnode    222.3050   26580.5895  force_treeallocate()/forcetree.c/4820
  48 2980 0            Father    221.9056   26802.4950  force_treeallocate()/forcetree.c/4827

Task=78: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.5553 MB).
task 78: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  70 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  70 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  70 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  70 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  70 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  70 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  48 2981 0           Ngblist     63.3448   26865.8399  subfind_density()/subfind_density.c/78
  48 2982 0         Dist2list    126.6897   26992.5296  subfind_density()/subfind_density.c/79
  48 2983 0    DataIndexTable     21.8182   27014.3478  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 78
  57 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  57 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  57 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  57 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  57 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  57 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  57 2973 0         FOF_PList    380.1491   18861.0012  fof_fof()/fof.c/249
  57 2974 0         FOF_GList    633.5818   19494.5830  fof_fof()/fof.c/337
  57 2975 0             Group     11.0203   19505.6033  fof_fof()/fof.c/380
  57 2976 0   DomainNodeIndex      0.3495   19505.9529  force_treeallocate()/forcetree.c/4800
  57 2977 0        Nodes_base   3737.6006   23243.5535  force_treeallocate()/forcetree.c/4803
  57 2978 0     Extnodes_base   3114.6672   26358.2207  force_treeallocate()/forcetree.c/4811
  57 2979 0          Nextnode    222.3050   26580.5257  force_treeallocate()/forcetree.c/4820
  70 2973 0         FOF_PList    379.8450   18860.6972  fof_fof()/fof.c/249
  70 2974 0         FOF_GList    633.0751   19493.7723  fof_fof()/fof.c/337
  70 2975 0             Group     10.3397   19504.1119  fof_fof()/fof.c/380
  70 2976 0   DomainNodeIndex      0.3495   19504.4615  force_treeallocate()/forcetree.c/4800
  70 2977 0        Nodes_base   3737.6006   23242.0621  force_treeallocate()/forcetree.c/4803
  70 2978 0     Extnodes_base   3114.6672   26356.7293  force_treeallocate()/forcetree.c/4811
  70 2979 0          Nextnode    222.3050   26579.0343  force_treeallocate()/forcetree.c/4820
  70 2980 0            Father    221.9056   26800.9399  force_treeallocate()/forcetree.c/4827
  70 2981 0           Ngblist     63.3075   26864.2474  subfind_density()/subfind_density.c/78
  70 2982 0         Dist2list    126.6150   26990.8625  subfind_density()/subfind_density.c/79
  70 2983 0    DataIndexTable     21.8182   27012.6806  subfind_density()/subfind_density.c/87
  70 2984 0      DataNodeList     58.1818   27070.8624  subfind_density()/subfind_density.c/89
  70 2985 0              Left    126.6150   27197.4774  subfind_density()/subfind_density.c/91
  70 2986 0             Right    126.6150   27324.0925  subfind_density()/subfind_density.c/92
  48 2984 0      DataNodeList     58.1818   27072.5296  subfind_density()/subfind_density.c/89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  57 2980 0            Father    221.9056   26802.4313  force_treeallocate()/forcetree.c/4827
  57 2981 0           Ngblist     63.3582   26865.7895  subfind_density()/subfind_density.c/78
  57 2982 0         Dist2list    126.7164   26992.5059  subfind_density()/subfind_density.c/79
  57 2983 0    DataIndexTable     21.8182   27014.3240  subfind_density()/subfind_density.c/87
  70 2987 0              Todo     15.8269   27339.9193  subfind_density()/subfind_density.c/93
  70 2988 0             DM_Vx    126.6150   27466.5343  subfind_density()/subfind_density.c/95
  70 2989 0             DM_Vy    126.6150   27593.1494  subfind_density()/subfind_density.c/96
  48 2985 0              Left    126.6897   27199.2192  subfind_density()/subfind_density.c/91
  79 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  79 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  79 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  79 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  79 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  79 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  79 2973 0         FOF_PList    380.6501   18861.5023  fof_fof()/fof.c/249
  79 2974 0         FOF_GList    634.4168   19495.9191  fof_fof()/fof.c/337
  79 2975 0             Group     10.3397   19506.2587  fof_fof()/fof.c/380
  79 2976 0   DomainNodeIndex      0.3495   19506.6082  force_treeallocate()/forcetree.c/4800
  79 2977 0        Nodes_base   3737.6006   23244.2089  force_treeallocate()/forcetree.c/4803
  79 2978 0     Extnodes_base   3114.6672   26358.8761  force_treeallocate()/forcetree.c/4811
  57 2984 0      DataNodeList     58.1818   27072.5058  subfind_density()/subfind_density.c/89
  57 2985 0              Left    126.7164   27199.2222  subfind_density()/subfind_density.c/91
  57 2986 0             Right    126.7164   27325.9385  subfind_density()/subfind_density.c/92
  70 2990 0             DM_Vz    126.6150   27719.7644  subfind_density()/subfind_density.c/97
  70 2991 0    	  DensDataGet    146.8738   27866.6382  subfind_density()/subfind_density.c/161
  70 2992 0  	  DensDataResult    110.1554   27976.7936  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  48 2986 0             Right    126.6897   27325.9089  subfind_density()/subfind_density.c/92
  79 2979 0          Nextnode    222.3050   26581.1811  force_treeallocate()/forcetree.c/4820
  79 2980 0            Father    221.9056   26803.0867  force_treeallocate()/forcetree.c/4827
  79 2981 0           Ngblist     63.4417   26866.5284  subfind_density()/subfind_density.c/78
  79 2982 0         Dist2list    126.8834   26993.4118  subfind_density()/subfind_density.c/79
  57 2987 0              Todo     15.8395   27341.7781  subfind_density()/subfind_density.c/93
  57 2988 0             DM_Vx    126.7164   27468.4944  subfind_density()/subfind_density.c/95
  57 2989 0             DM_Vy    126.7164   27595.2108  subfind_density()/subfind_density.c/96
  57 2990 0             DM_Vz    126.7164   27721.9271  subfind_density()/subfind_density.c/97

Task=70: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=23.2064 MB).
task 70: endrun called with an error level of 812


  48 2987 0              Todo     15.8362   27341.7451  subfind_density()/subfind_density.c/93
  79 2983 0    DataIndexTable     21.8182   27015.2299  subfind_density()/subfind_density.c/87
  79 2984 0      DataNodeList     58.1818   27073.4117  subfind_density()/subfind_density.c/89
  79 2985 0              Left    126.8834   27200.2951  subfind_density()/subfind_density.c/91
  79 2986 0             Right    126.8834   27327.1785  subfind_density()/subfind_density.c/92
  57 2991 0    	  DensDataGet    121.7355   27843.6626  subfind_density()/subfind_density.c/161
  57 2992 0  	  DensDataResult     91.3016   27934.9642  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=57: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.0358 MB).
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 70
  48 2988 0             DM_Vx    126.6897   27468.4348  subfind_density()/subfind_density.c/95
  79 2987 0              Todo     15.8604   27343.0389  subfind_density()/subfind_density.c/93
  79 2988 0             DM_Vx    126.8834   27469.9222  subfind_density()/subfind_density.c/95
  79 2989 0             DM_Vy    126.8834   27596.8056  subfind_density()/subfind_density.c/96
task 57: endrun called with an error level of 812


  48 2989 0             DM_Vy    126.6897   27595.1245  subfind_density()/subfind_density.c/96
  79 2990 0             DM_Vz    126.8834   27723.6890  subfind_density()/subfind_density.c/97
  79 2991 0    	  DensDataGet    122.8380   27846.5270  subfind_density()/subfind_density.c/161
  79 2992 0  	  DensDataResult     92.1285   27938.6555  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 57
  48 2990 0             DM_Vz    126.6897   27721.8142  subfind_density()/subfind_density.c/97

  48 2991 0    	  DensDataGet    121.2556   27843.0698  subfind_density()/subfind_density.c/161

Task=79: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.3445 MB).
task 79: endrun called with an error level of 812


  48 2992 0  	  DensDataResult     90.9417   27934.0115  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 79

Task=48: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.9885 MB).
task 48: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 48
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 An HDF5 error was detected, we'd better stop.
  64 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  64 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  64 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  64 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  64 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  64 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  64 2973 0         FOF_PList    380.5051   18861.3573  fof_fof()/fof.c/249
  64 2974 0         FOF_GList    634.1752   19495.5325  fof_fof()/fof.c/337
  64 2975 0             Group     10.3905   19505.9230  fof_fof()/fof.c/380
  64 2976 0   DomainNodeIndex      0.3495   19506.2725  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
HDF5-DIAG: Error detected in HDF5 (1.8.12)   64 2977 0        Nodes_base   3737.6006   23243.8731  force_treeallocate()/forcetree.c/4803
  64 2978 0     Extnodes_base   3114.6672   26358.5403  force_treeallocate()/forcetree.c/4811
  64 2979 0          Nextnode    222.3050   26580.8454  force_treeallocate()/forcetree.c/4820
  63 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  63 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  63 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  63 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  63 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  63 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  63 2973 0         FOF_PList    380.6473   18861.4995  fof_fof()/fof.c/249
  63 2974 0         FOF_GList    634.4122   19495.9117  fof_fof()/fof.c/337
  63 2975 0             Group     10.8128   19506.7245  fof_fof()/fof.c/380
  63 2976 0   DomainNodeIndex      0.3495   19507.0741  force_treeallocate()/forcetree.c/4800
  63 2977 0        Nodes_base   3737.6006   23244.6747  force_treeallocate()/forcetree.c/4803
  63 2978 0     Extnodes_base   3114.6672   26359.3419  force_treeallocate()/forcetree.c/4811
  63 2979 0          Nextnode    222.3050   26581.6470  force_treeallocate()/forcetree.c/4820
  63 2980 0            Father    221.9056   26803.5525  force_treeallocate()/forcetree.c/4827
  63 2981 0           Ngblist     63.4412   26866.9938  subfind_density()/subfind_density.c/78
task 48: endrun called with an error level of 124


  64 2980 0            Father    221.9056   26802.7510  force_treeallocate()/forcetree.c/4827
  64 2981 0           Ngblist     63.4175   26866.1685  subfind_density()/subfind_density.c/78
  64 2982 0         Dist2list    126.8350   26993.0035  subfind_density()/subfind_density.c/79
  63 2982 0         Dist2list    126.8824   26993.8762  subfind_density()/subfind_density.c/79
  63 2983 0    DataIndexTable     21.8182   27015.6944  subfind_density()/subfind_density.c/87
  63 2984 0      DataNodeList     58.1818   27073.8762  subfind_density()/subfind_density.c/89
  63 2985 0              Left    126.8824   27200.7586  subfind_density()/subfind_density.c/91
  63 2986 0             Right    126.8824   27327.6411  subfind_density()/subfind_density.c/92
MPI-process 48:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 48
  64 2983 0    DataIndexTable     21.8182   27014.8217  subfind_density()/subfind_density.c/87
  64 2984 0      DataNodeList     58.1818   27073.0035  subfind_density()/subfind_density.c/89
  64 2985 0              Left    126.8350   27199.8385  subfind_density()/subfind_density.c/91
  64 2986 0             Right    126.8350   27326.6736  subfind_density()/subfind_density.c/92
  63 2987 0              Todo     15.8603   27343.5014  subfind_density()/subfind_density.c/93
  63 2988 0             DM_Vx    126.8824   27470.3838  subfind_density()/subfind_density.c/95
  63 2989 0             DM_Vy    126.8824   27597.2663  subfind_density()/subfind_density.c/96
  63 2990 0             DM_Vz    126.8824   27724.1487  subfind_density()/subfind_density.c/97
  64 2987 0              Todo     15.8544   27342.5280  subfind_density()/subfind_density.c/93
  64 2988 0             DM_Vx    126.8350   27469.3630  subfind_density()/subfind_density.c/95
  64 2989 0             DM_Vy    126.8350   27596.1980  subfind_density()/subfind_density.c/96
  63 2991 0    	  DensDataGet    124.3161   27848.4648  subfind_density()/subfind_density.c/161
  63 2992 0  	  DensDataResult     93.2371   27941.7019  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  64 2990 0             DM_Vz    126.8350   27723.0331  subfind_density()/subfind_density.c/97

Task=63: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.2981 MB).
task 63: endrun called with an error level of 812


  64 2991 0    	  DensDataGet    134.2487   27857.2818  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 63
  64 2992 0  	  DensDataResult    100.6865   27957.9683  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=64: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.0317 MB).
task 64: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 64
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 64:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
task 64: endrun called with an error level of 124


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  74 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  74 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  74 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  74 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  74 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  74 2973 0         FOF_PList    380.5471   18861.3993  fof_fof()/fof.c/249
  74 2974 0         FOF_GList    634.2452   19495.6445  fof_fof()/fof.c/337
  74 2975 0             Group     11.2393   19506.8838  fof_fof()/fof.c/380
  74 2976 0   DomainNodeIndex      0.3495   19507.2333  force_treeallocate()/forcetree.c/4800
  74 2977 0        Nodes_base   3737.6006   23244.8340  force_treeallocate()/forcetree.c/4803
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 64
  53 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  53 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  53 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  53 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  53 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  53 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  53 2973 0         FOF_PList    380.3017   18861.1538  fof_fof()/fof.c/249
  53 2974 0         FOF_GList    633.8361   19494.9899  fof_fof()/fof.c/337
  53 2975 0             Group     11.0577   19506.0476  fof_fof()/fof.c/380
  53 2976 0   DomainNodeIndex      0.3495   19506.3971  force_treeallocate()/forcetree.c/4800
  53 2977 0        Nodes_base   3737.6006   23243.9978  force_treeallocate()/forcetree.c/4803
  53 2978 0     Extnodes_base   3114.6672   26358.6650  force_treeallocate()/forcetree.c/4811
  53 2979 0          Nextnode    222.3050   26580.9700  force_treeallocate()/forcetree.c/4820
  53 2980 0            Father    221.9056   26802.8756  force_treeallocate()/forcetree.c/4827
  53 2981 0           Ngblist     63.3836   26866.2592  subfind_density()/subfind_density.c/78
  53 2982 0         Dist2list    126.7672   26993.0264  subfind_density()/subfind_density.c/79
  74 2978 0     Extnodes_base   3114.6672   26359.5012  force_treeallocate()/forcetree.c/4811
  74 2979 0          Nextnode    222.3050   26581.8062  force_treeallocate()/forcetree.c/4820
  53 2983 0    DataIndexTable     21.8182   27014.8446  subfind_density()/subfind_density.c/87
  53 2984 0      DataNodeList     58.1818   27073.0264  subfind_density()/subfind_density.c/89
  53 2985 0              Left    126.7672   27199.7936  subfind_density()/subfind_density.c/91
  53 2986 0             Right    126.7672   27326.5608  subfind_density()/subfind_density.c/92
  53 2987 0              Todo     15.8459   27342.4068  subfind_density()/subfind_density.c/93
  74 2980 0            Father    221.9056   26803.7118  force_treeallocate()/forcetree.c/4827
  74 2981 0           Ngblist     63.4245   26867.1363  subfind_density()/subfind_density.c/78
  74 2982 0         Dist2list    126.8490   26993.9854  subfind_density()/subfind_density.c/79
  53 2988 0             DM_Vx    126.7672   27469.1740  subfind_density()/subfind_density.c/95
  53 2989 0             DM_Vy    126.7672   27595.9412  subfind_density()/subfind_density.c/96
  53 2990 0             DM_Vz    126.7672   27722.7084  subfind_density()/subfind_density.c/97
  53 2991 0    	  DensDataGet    121.8845   27844.5929  subfind_density()/subfind_density.c/161
  74 2983 0    DataIndexTable     21.8182   27015.8035  subfind_density()/subfind_density.c/87
  74 2984 0      DataNodeList     58.1818   27073.9853  subfind_density()/subfind_density.c/89
  74 2985 0              Left    126.8490   27200.8344  subfind_density()/subfind_density.c/91
  53 2992 0  	  DensDataResult     91.4134   27936.0063  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=53: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=63.9937 MB).
task 53: endrun called with an error level of 812


  74 2986 0             Right    126.8490   27327.6834  subfind_density()/subfind_density.c/92
  74 2987 0              Todo     15.8561   27343.5395  subfind_density()/subfind_density.c/93
  74 2988 0             DM_Vx    126.8490   27470.3886  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 53
  74 2989 0             DM_Vy    126.8490   27597.2376  subfind_density()/subfind_density.c/96
  74 2990 0             DM_Vz    126.8490   27724.0867  subfind_density()/subfind_density.c/97
  74 2991 0    	  DensDataGet    119.0290   27843.1157  subfind_density()/subfind_density.c/161
  74 2992 0  	  DensDataResult     89.2717   27932.3874  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=74: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.6126 MB).
task 74: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 74
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  65 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  65 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  65 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  65 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  65 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  65 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  65 2973 0         FOF_PList    380.3223   18861.1745  fof_fof()/fof.c/249
  65 2974 0         FOF_GList    633.8706   19495.0451  fof_fof()/fof.c/337
  65 2975 0             Group     10.3397   19505.3848  fof_fof()/fof.c/380
  65 2976 0   DomainNodeIndex      0.3495   19505.7343  force_treeallocate()/forcetree.c/4800
  65 2977 0        Nodes_base   3737.6006   23243.3349  force_treeallocate()/forcetree.c/4803
  65 2978 0     Extnodes_base   3114.6672   26358.0021  force_treeallocate()/forcetree.c/4811
  65 2979 0          Nextnode    222.3050   26580.3072  force_treeallocate()/forcetree.c/4820
  62 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  62 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  62 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  62 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  62 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  62 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  65 2980 0            Father    221.9056   26802.2128  force_treeallocate()/forcetree.c/4827
  65 2981 0           Ngblist     63.3871   26865.5998  subfind_density()/subfind_density.c/78
  65 2982 0         Dist2list    126.7741   26992.3739  subfind_density()/subfind_density.c/79
  65 2983 0    DataIndexTable     21.8182   27014.1921  subfind_density()/subfind_density.c/87
  62 2973 0         FOF_PList    380.6150   18861.4672  fof_fof()/fof.c/249
  62 2974 0         FOF_GList    634.3584   19495.8256  fof_fof()/fof.c/337
  62 2975 0             Group     12.3806   19508.2062  fof_fof()/fof.c/380
  65 2984 0      DataNodeList     58.1818   27072.3739  subfind_density()/subfind_density.c/89
  65 2985 0              Left    126.7741   27199.1480  subfind_density()/subfind_density.c/91
  65 2986 0             Right    126.7741   27325.9221  subfind_density()/subfind_density.c/92
  65 2987 0              Todo     15.8468   27341.7689  subfind_density()/subfind_density.c/93
  62 2976 0   DomainNodeIndex      0.3495   19508.5558  force_treeallocate()/forcetree.c/4800
  62 2977 0        Nodes_base   3737.6006   23246.1564  force_treeallocate()/forcetree.c/4803
  62 2978 0     Extnodes_base   3114.6672   26360.8236  force_treeallocate()/forcetree.c/4811
  65 2988 0             DM_Vx    126.7741   27468.5430  subfind_density()/subfind_density.c/95
  65 2989 0             DM_Vy    126.7741   27595.3171  subfind_density()/subfind_density.c/96
  65 2990 0             DM_Vz    126.7741   27722.0913  subfind_density()/subfind_density.c/97
  62 2979 0          Nextnode    222.3050   26583.1286  force_treeallocate()/forcetree.c/4820
  62 2980 0            Father    221.9056   26805.0342  force_treeallocate()/forcetree.c/4827
  62 2981 0           Ngblist     63.4358   26868.4701  subfind_density()/subfind_density.c/78
  65 2991 0    	  DensDataGet    110.1943   27832.2855  subfind_density()/subfind_density.c/161
  65 2992 0  	  DensDataResult     82.6457   27914.9312  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=65: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.0688 MB).
  62 2982 0         Dist2list    126.8717   26995.3418  subfind_density()/subfind_density.c/79
  62 2983 0    DataIndexTable     21.8182   27017.1599  subfind_density()/subfind_density.c/87
  62 2984 0      DataNodeList     58.1818   27075.3417  subfind_density()/subfind_density.c/89
task 65: endrun called with an error level of 812


  62 2985 0              Left    126.8717   27202.2134  subfind_density()/subfind_density.c/91
  62 2986 0             Right    126.8717   27329.0851  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 65
  62 2987 0              Todo     15.8590   27344.9441  subfind_density()/subfind_density.c/93
  62 2988 0             DM_Vx    126.8717   27471.8157  subfind_density()/subfind_density.c/95
  62 2989 0             DM_Vy    126.8717   27598.6874  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  62 2990 0             DM_Vz    126.8717   27725.5591  subfind_density()/subfind_density.c/97
  52 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  52 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  52 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  52 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  52 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  52 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  52 2973 0         FOF_PList    380.3748   18861.2270  fof_fof()/fof.c/249
  52 2974 0         FOF_GList    633.9581   19495.1851  fof_fof()/fof.c/337
  52 2975 0             Group     10.3397   19505.5247  fof_fof()/fof.c/380
  52 2976 0   DomainNodeIndex      0.3495   19505.8742  force_treeallocate()/forcetree.c/4800
  75 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  75 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  75 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  75 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  75 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  75 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  75 2973 0         FOF_PList    380.2752   18861.1273  fof_fof()/fof.c/249
  75 2974 0         FOF_GList    633.7920   19494.9193  fof_fof()/fof.c/337
  75 2975 0             Group     11.6232   19506.5425  fof_fof()/fof.c/380
  75 2976 0   DomainNodeIndex      0.3495   19506.8920  force_treeallocate()/forcetree.c/4800
  75 2977 0        Nodes_base   3737.6006   23244.4926  force_treeallocate()/forcetree.c/4803
  75 2978 0     Extnodes_base   3114.6672   26359.1599  force_treeallocate()/forcetree.c/4811
  62 2991 0    	  DensDataGet    152.5667   27878.1258  subfind_density()/subfind_density.c/161
  52 2977 0        Nodes_base   3737.6006   23243.4749  force_treeallocate()/forcetree.c/4803
  52 2978 0     Extnodes_base   3114.6672   26358.1421  force_treeallocate()/forcetree.c/4811
  52 2979 0          Nextnode    222.3050   26580.4471  force_treeallocate()/forcetree.c/4820
  75 2979 0          Nextnode    222.3050   26581.4649  force_treeallocate()/forcetree.c/4820
  75 2980 0            Father    221.9056   26803.3705  force_treeallocate()/forcetree.c/4827
  75 2981 0           Ngblist     63.3792   26866.7497  subfind_density()/subfind_density.c/78
  75 2982 0         Dist2list    126.7584   26993.5081  subfind_density()/subfind_density.c/79
  75 2983 0    DataIndexTable     21.8182   27015.3262  subfind_density()/subfind_density.c/87
  62 2992 0  	  DensDataResult    114.4250   27992.5508  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  52 2980 0            Father    221.9056   26802.3527  force_treeallocate()/forcetree.c/4827
  52 2981 0           Ngblist     63.3958   26865.7485  subfind_density()/subfind_density.c/78
  52 2982 0         Dist2list    126.7916   26992.5401  subfind_density()/subfind_density.c/79
  75 2984 0      DataNodeList     58.1818   27073.5080  subfind_density()/subfind_density.c/89
  75 2985 0              Left    126.7584   27200.2664  subfind_density()/subfind_density.c/91
  75 2986 0             Right    126.7584   27327.0248  subfind_density()/subfind_density.c/92

Task=62: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=7.44916 MB).
task 62: endrun called with an error level of 812


  52 2983 0    DataIndexTable     21.8182   27014.3583  subfind_density()/subfind_density.c/87
  52 2984 0      DataNodeList     58.1818   27072.5401  subfind_density()/subfind_density.c/89
  52 2985 0              Left    126.7916   27199.3317  subfind_density()/subfind_density.c/91
  75 2987 0              Todo     15.8448   27342.8696  subfind_density()/subfind_density.c/93
  75 2988 0             DM_Vx    126.7584   27469.6280  subfind_density()/subfind_density.c/95
  75 2989 0             DM_Vy    126.7584   27596.3864  subfind_density()/subfind_density.c/96
  75 2990 0             DM_Vz    126.7584   27723.1448  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 62
  52 2986 0             Right    126.7916   27326.1233  subfind_density()/subfind_density.c/92
  52 2987 0              Todo     15.8490   27341.9723  subfind_density()/subfind_density.c/93
  52 2988 0             DM_Vx    126.7916   27468.7639  subfind_density()/subfind_density.c/95
  75 2991 0    	  DensDataGet    118.4672   27841.6120  subfind_density()/subfind_density.c/161
  75 2992 0  	  DensDataResult     88.8504   27930.4623  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  52 2989 0             DM_Vy    126.7916   27595.5555  subfind_density()/subfind_density.c/96
  52 2990 0             DM_Vz    126.7916   27722.3471  subfind_density()/subfind_density.c/97
  52 2991 0    	  DensDataGet    110.5390   27832.8861  subfind_density()/subfind_density.c/161

Task=75: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.5377 MB).
task 75: endrun called with an error level of 812


  56 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  56 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  56 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  56 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  56 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  56 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  56 2973 0         FOF_PList    380.2502   18861.1024  fof_fof()/fof.c/249
  71 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  71 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  71 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  71 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  71 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  71 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  71 2973 0         FOF_PList    380.6329   18861.4851  fof_fof()/fof.c/249
  71 2974 0         FOF_GList    634.3882   19495.8734  fof_fof()/fof.c/337
  71 2975 0             Group     10.3397   19506.2130  fof_fof()/fof.c/380
  71 2976 0   DomainNodeIndex      0.3495   19506.5625  force_treeallocate()/forcetree.c/4800
  71 2977 0        Nodes_base   3737.6006   23244.1632  force_treeallocate()/forcetree.c/4803
  71 2978 0     Extnodes_base   3114.6672   26358.8304  force_treeallocate()/forcetree.c/4811
  52 2992 0  	  DensDataResult     82.9043   27915.7904  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 75
  56 2974 0         FOF_GList    633.7503   19494.8527  fof_fof()/fof.c/337
  56 2975 0             Group     10.7246   19505.5773  fof_fof()/fof.c/380
  56 2976 0   DomainNodeIndex      0.3495   19505.9268  force_treeallocate()/forcetree.c/4800
  71 2979 0          Nextnode    222.3050   26581.1354  force_treeallocate()/forcetree.c/4820
  71 2980 0            Father    221.9056   26803.0410  force_treeallocate()/forcetree.c/4827
  71 2981 0           Ngblist     63.4388   26866.4798  subfind_density()/subfind_density.c/78
  71 2982 0         Dist2list    126.8776   26993.3575  subfind_density()/subfind_density.c/79

  56 2977 0        Nodes_base   3737.6006   23243.5275  force_treeallocate()/forcetree.c/4803
  56 2978 0     Extnodes_base   3114.6672   26358.1947  force_treeallocate()/forcetree.c/4811
  71 2983 0    DataIndexTable     21.8182   27015.1757  subfind_density()/subfind_density.c/87
  71 2984 0      DataNodeList     58.1818   27073.3575  subfind_density()/subfind_density.c/89
  71 2985 0              Left    126.8776   27200.2351  subfind_density()/subfind_density.c/91

Task=52: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.2096 MB).
task 52: endrun called with an error level of 812


  56 2979 0          Nextnode    222.3050   26580.4997  force_treeallocate()/forcetree.c/4820
  56 2980 0            Father    221.9056   26802.4053  force_treeallocate()/forcetree.c/4827
  56 2981 0           Ngblist     63.3750   26865.7804  subfind_density()/subfind_density.c/78
  71 2986 0             Right    126.8776   27327.1127  subfind_density()/subfind_density.c/92
  71 2987 0              Todo     15.8597   27342.9725  subfind_density()/subfind_density.c/93
  71 2988 0             DM_Vx    126.8776   27469.8501  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 52
  56 2982 0         Dist2list    126.7501   26992.5304  subfind_density()/subfind_density.c/79
  56 2983 0    DataIndexTable     21.8182   27014.3486  subfind_density()/subfind_density.c/87
  56 2984 0      DataNodeList     58.1818   27072.5304  subfind_density()/subfind_density.c/89
  71 2989 0             DM_Vy    126.8776   27596.7278  subfind_density()/subfind_density.c/96
  71 2990 0             DM_Vz    126.8776   27723.6054  subfind_density()/subfind_density.c/97
  71 2991 0    	  DensDataGet    108.2916   27831.8970  subfind_density()/subfind_density.c/161
  72 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  72 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  72 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  72 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  72 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  72 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  72 2973 0         FOF_PList    380.6348   18861.4869  fof_fof()/fof.c/249
  72 2974 0         FOF_GList    634.3913   19495.8782  fof_fof()/fof.c/337
  72 2975 0             Group     13.0001   19508.8782  fof_fof()/fof.c/380
  72 2976 0   DomainNodeIndex      0.3495   19509.2278  force_treeallocate()/forcetree.c/4800
  72 2977 0        Nodes_base   3737.6006   23246.8284  force_treeallocate()/forcetree.c/4803
  56 2985 0              Left    126.7501   27199.2805  subfind_density()/subfind_density.c/91
  56 2986 0             Right    126.7501   27326.0305  subfind_density()/subfind_density.c/92
  71 2992 0  	  DensDataResult     81.2187   27913.1157  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=71: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.8843 MB).
task 71: endrun called with an error level of 812


  72 2978 0     Extnodes_base   3114.6672   26361.4956  force_treeallocate()/forcetree.c/4811
  72 2979 0          Nextnode    222.3050   26583.8006  force_treeallocate()/forcetree.c/4820
  56 2987 0              Todo     15.8438   27341.8743  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 71
  72 2980 0            Father    221.9056   26805.7062  force_treeallocate()/forcetree.c/4827
  72 2981 0           Ngblist     63.4391   26869.1454  subfind_density()/subfind_density.c/78
  72 2982 0         Dist2list    126.8783   26996.0236  subfind_density()/subfind_density.c/79
  72 2983 0    DataIndexTable     21.8182   27017.8418  subfind_density()/subfind_density.c/87
  56 2988 0             DM_Vx    126.7501   27468.6244  subfind_density()/subfind_density.c/95
  72 2984 0      DataNodeList     58.1818   27076.0236  subfind_density()/subfind_density.c/89
  72 2985 0              Left    126.8783   27202.9018  subfind_density()/subfind_density.c/91
  72 2986 0             Right    126.8783   27329.7801  subfind_density()/subfind_density.c/92
  56 2989 0             DM_Vy    126.7501   27595.3744  subfind_density()/subfind_density.c/96
  72 2987 0              Todo     15.8598   27345.6399  subfind_density()/subfind_density.c/93
  72 2988 0             DM_Vx    126.8783   27472.5181  subfind_density()/subfind_density.c/95
  56 2990 0             DM_Vz    126.7501   27722.1245  subfind_density()/subfind_density.c/97
  72 2989 0             DM_Vy    126.8783   27599.3964  subfind_density()/subfind_density.c/96
  56 2991 0    	  DensDataGet    124.7141   27846.8386  subfind_density()/subfind_density.c/161
  72 2990 0             DM_Vz    126.8783   27726.2746  subfind_density()/subfind_density.c/97
  56 2992 0  	  DensDataResult     93.5356   27940.3742  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  72 2991 0    	  DensDataGet    114.7218   27840.9964  subfind_density()/subfind_density.c/161

Task=56: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.6258 MB).
  72 2992 0  	  DensDataResult     86.0414   27927.0378  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 56: endrun called with an error level of 812



Task=72: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.9622 MB).
task 72: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 56
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 72
 An HDF5 error was detected, we'd better stop.
task 72: endrun called with an error level of 124


HDF5-DIAG: Error detected in HDF5 (1.8.12)  An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 72:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 72
MPI-process 56:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 56: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 56
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  60 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  60 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  60 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  60 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  60 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  60 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  60 2973 0         FOF_PList    380.0810   18860.9331  fof_fof()/fof.c/249
  60 2974 0         FOF_GList    633.4683   19494.4014  fof_fof()/fof.c/337
  60 2975 0             Group     13.4981   19507.8995  fof_fof()/fof.c/380
  60 2976 0   DomainNodeIndex      0.3495   19508.2491  force_treeallocate()/forcetree.c/4800
  60 2977 0        Nodes_base   3737.6006   23245.8497  force_treeallocate()/forcetree.c/4803
  60 2978 0     Extnodes_base   3114.6672   26360.5169  force_treeallocate()/forcetree.c/4811
  60 2979 0          Nextnode    222.3050   26582.8219  force_treeallocate()/forcetree.c/4820
  60 2980 0            Father    221.9056   26804.7275  force_treeallocate()/forcetree.c/4827
  60 2981 0           Ngblist     63.3468   26868.0744  subfind_density()/subfind_density.c/78
  60 2982 0         Dist2list    126.6937   26994.7680  subfind_density()/subfind_density.c/79
  60 2983 0    DataIndexTable     21.8182   27016.5862  subfind_density()/subfind_density.c/87
  60 2984 0      DataNodeList     58.1818   27074.7680  subfind_density()/subfind_density.c/89
  60 2985 0              Left    126.6937   27201.4616  subfind_density()/subfind_density.c/91
  60 2986 0             Right    126.6937   27328.1553  subfind_density()/subfind_density.c/92
  60 2987 0              Todo     15.8367   27343.9920  subfind_density()/subfind_density.c/93
  60 2988 0             DM_Vx    126.6937   27470.6857  subfind_density()/subfind_density.c/95
  60 2989 0             DM_Vy    126.6937   27597.3793  subfind_density()/subfind_density.c/96
  60 2990 0             DM_Vz    126.6937   27724.0730  subfind_density()/subfind_density.c/97
  60 2991 0    	  DensDataGet    109.1923   27833.2653  subfind_density()/subfind_density.c/161
  60 2992 0  	  DensDataResult     81.8942   27915.1595  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=60: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.8405 MB).
task 60: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 60
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  51 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  51 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  51 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  51 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  51 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  51 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  51 2973 0         FOF_PList    379.8065   18860.6587  fof_fof()/fof.c/249
  51 2974 0         FOF_GList    633.0109   19493.6696  fof_fof()/fof.c/337
  51 2975 0             Group     10.6572   19504.3267  fof_fof()/fof.c/380
  51 2976 0   DomainNodeIndex      0.3495   19504.6762  force_treeallocate()/forcetree.c/4800
  51 2977 0        Nodes_base   3737.6006   23242.2769  force_treeallocate()/forcetree.c/4803
  51 2978 0     Extnodes_base   3114.6672   26356.9441  force_treeallocate()/forcetree.c/4811
  51 2979 0          Nextnode    222.3050   26579.2491  force_treeallocate()/forcetree.c/4820
  51 2980 0            Father    221.9056   26801.1547  force_treeallocate()/forcetree.c/4827
  51 2981 0           Ngblist     63.3011   26864.4558  subfind_density()/subfind_density.c/78
  76 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  76 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  76 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  76 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  76 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  76 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  76 2973 0         FOF_PList    380.5325   18861.3847  fof_fof()/fof.c/249
  76 2974 0         FOF_GList    634.2209   19495.6056  fof_fof()/fof.c/337
  76 2975 0             Group     11.4198   19507.0254  fof_fof()/fof.c/380
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  51 2982 0         Dist2list    126.6022   26991.0580  subfind_density()/subfind_density.c/79
  51 2983 0    DataIndexTable     21.8182   27012.8762  subfind_density()/subfind_density.c/87
  51 2984 0      DataNodeList     58.1818   27071.0579  subfind_density()/subfind_density.c/89
  51 2985 0              Left    126.6022   27197.6601  subfind_density()/subfind_density.c/91
  51 2986 0             Right    126.6022   27324.2623  subfind_density()/subfind_density.c/92
  76 2976 0   DomainNodeIndex      0.3495   19507.3749  force_treeallocate()/forcetree.c/4800
  76 2977 0        Nodes_base   3737.6006   23244.9756  force_treeallocate()/forcetree.c/4803
  69 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  69 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  69 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  69 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  69 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  69 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  69 2973 0         FOF_PList    380.5331   18861.3852  fof_fof()/fof.c/249
  69 2974 0         FOF_GList    634.2218   19495.6070  fof_fof()/fof.c/337
  69 2975 0             Group     12.2644   19507.8714  fof_fof()/fof.c/380
  69 2976 0   DomainNodeIndex      0.3495   19508.2209  force_treeallocate()/forcetree.c/4800
  69 2977 0        Nodes_base   3737.6006   23245.8216  force_treeallocate()/forcetree.c/4803
  69 2978 0     Extnodes_base   3114.6672   26360.4888  force_treeallocate()/forcetree.c/4811
  69 2979 0          Nextnode    222.3050   26582.7938  force_treeallocate()/forcetree.c/4820
  69 2980 0            Father    221.9056   26804.6994  force_treeallocate()/forcetree.c/4827
  69 2981 0           Ngblist     63.4222   26868.1216  subfind_density()/subfind_density.c/78
  69 2982 0         Dist2list    126.8444   26994.9659  subfind_density()/subfind_density.c/79
  58 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  58 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  58 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  58 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  58 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  58 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  58 2973 0         FOF_PList    380.6336   18861.4858  fof_fof()/fof.c/249
  58 2974 0         FOF_GList    634.3894   19495.8752  fof_fof()/fof.c/337
  51 2987 0              Todo     15.8253   27340.0876  subfind_density()/subfind_density.c/93
  51 2988 0             DM_Vx    126.6022   27466.6897  subfind_density()/subfind_density.c/95
  51 2989 0             DM_Vy    126.6022   27593.2919  subfind_density()/subfind_density.c/96
  76 2978 0     Extnodes_base   3114.6672   26359.6428  force_treeallocate()/forcetree.c/4811
  76 2979 0          Nextnode    222.3050   26581.9478  force_treeallocate()/forcetree.c/4820
  69 2983 0    DataIndexTable     21.8182   27016.7841  subfind_density()/subfind_density.c/87
  69 2984 0      DataNodeList     58.1818   27074.9659  subfind_density()/subfind_density.c/89
  69 2985 0              Left    126.8444   27201.8102  subfind_density()/subfind_density.c/91
  69 2986 0             Right    126.8444   27328.6546  subfind_density()/subfind_density.c/92
  69 2987 0              Todo     15.8555   27344.5101  subfind_density()/subfind_density.c/93
  58 2975 0             Group     10.3397   19506.2148  fof_fof()/fof.c/380
  58 2976 0   DomainNodeIndex      0.3495   19506.5644  force_treeallocate()/forcetree.c/4800
  51 2990 0             DM_Vz    126.6022   27719.8941  subfind_density()/subfind_density.c/97
  51 2991 0    	  DensDataGet    115.9807   27835.8748  subfind_density()/subfind_density.c/161
  51 2992 0  	  DensDataResult     86.9855   27922.8603  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  76 2980 0            Father    221.9056   26803.8534  force_treeallocate()/forcetree.c/4827
  76 2981 0           Ngblist     63.4221   26867.2755  subfind_density()/subfind_density.c/78
  69 2988 0             DM_Vx    126.8444   27471.3545  subfind_density()/subfind_density.c/95
  69 2989 0             DM_Vy    126.8444   27598.1989  subfind_density()/subfind_density.c/96
  69 2990 0             DM_Vz    126.8444   27725.0432  subfind_density()/subfind_density.c/97
  58 2977 0        Nodes_base   3737.6006   23244.1650  force_treeallocate()/forcetree.c/4803
  58 2978 0     Extnodes_base   3114.6672   26358.8322  force_treeallocate()/forcetree.c/4811
  58 2979 0          Nextnode    222.3050   26581.1373  force_treeallocate()/forcetree.c/4820


Task=51: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.1397 MB).
task 51: endrun called with an error level of 812


  76 2982 0         Dist2list    126.8442   26994.1197  subfind_density()/subfind_density.c/79
  76 2983 0    DataIndexTable     21.8182   27015.9378  subfind_density()/subfind_density.c/87
  69 2991 0    	  DensDataGet    121.0866   27846.1298  subfind_density()/subfind_density.c/161
  69 2992 0  	  DensDataResult     90.8150   27936.9448  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=69: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=63.0552 MB).
  58 2980 0            Father    221.9056   26803.0428  force_treeallocate()/forcetree.c/4827
  58 2981 0           Ngblist     63.4389   26866.4818  subfind_density()/subfind_density.c/78
  58 2982 0         Dist2list    126.8779   26993.3597  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 51
  76 2984 0      DataNodeList     58.1818   27074.1196  subfind_density()/subfind_density.c/89
  76 2985 0              Left    126.8442   27200.9638  subfind_density()/subfind_density.c/91
task 69: endrun called with an error level of 812


  58 2983 0    DataIndexTable     21.8182   27015.1778  subfind_density()/subfind_density.c/87
  58 2984 0      DataNodeList     58.1818   27073.3596  subfind_density()/subfind_density.c/89
  58 2985 0              Left    126.8779   27200.2375  subfind_density()/subfind_density.c/91
  76 2986 0             Right    126.8442   27327.8080  subfind_density()/subfind_density.c/92
  76 2987 0              Todo     15.8555   27343.6635  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 69
  58 2986 0             Right    126.8779   27327.1154  subfind_density()/subfind_density.c/92
  76 2988 0             DM_Vx    126.8442   27470.5077  subfind_density()/subfind_density.c/95
  58 2987 0              Todo     15.8597   27342.9751  subfind_density()/subfind_density.c/93
  76 2989 0             DM_Vy    126.8442   27597.3519  subfind_density()/subfind_density.c/96
  58 2988 0             DM_Vx    126.8779   27469.8530  subfind_density()/subfind_density.c/95
  76 2990 0             DM_Vz    126.8442   27724.1960  subfind_density()/subfind_density.c/97
  58 2989 0             DM_Vy    126.8779   27596.7309  subfind_density()/subfind_density.c/96
  76 2991 0    	  DensDataGet    119.1934   27843.3894  subfind_density()/subfind_density.c/161
  58 2990 0             DM_Vz    126.8779   27723.6087  subfind_density()/subfind_density.c/97
  76 2992 0  	  DensDataResult     89.3950   27932.7844  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  58 2991 0    	  DensDataGet    117.3208   27840.9296  subfind_density()/subfind_density.c/161

Task=76: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.2156 MB).
task 76: endrun called with an error level of 812


  58 2992 0  	  DensDataResult     87.9906   27928.9202  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 76

Task=58: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.0798 MB).
task 58: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 58
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  68 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  68 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  68 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  68 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  68 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  68 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  68 2973 0         FOF_PList    380.5719   18861.4241  fof_fof()/fof.c/249
  68 2974 0         FOF_GList    634.2865   19495.7106  fof_fof()/fof.c/337
  68 2975 0             Group     11.7300   19507.4407  fof_fof()/fof.c/380
  68 2976 0   DomainNodeIndex      0.3495   19507.7902  force_treeallocate()/forcetree.c/4800
  68 2977 0        Nodes_base   3737.6006   23245.3908  force_treeallocate()/forcetree.c/4803
  68 2978 0     Extnodes_base   3114.6672   26360.0580  force_treeallocate()/forcetree.c/4811
  68 2979 0          Nextnode    222.3050   26582.3631  force_treeallocate()/forcetree.c/4820
  68 2980 0            Father    221.9056   26804.2687  force_treeallocate()/forcetree.c/4827
  68 2981 0           Ngblist     63.4287   26867.6973  subfind_density()/subfind_density.c/78
  68 2982 0         Dist2list    126.8573   26994.5546  subfind_density()/subfind_density.c/79
  68 2983 0    DataIndexTable     21.8182   27016.3728  subfind_density()/subfind_density.c/87
  68 2984 0      DataNodeList     58.1818   27074.5546  subfind_density()/subfind_density.c/89
  68 2985 0              Left    126.8573   27201.4119  subfind_density()/subfind_density.c/91
  68 2986 0             Right    126.8573   27328.2692  subfind_density()/subfind_density.c/92
  68 2987 0              Todo     15.8572   27344.1264  subfind_density()/subfind_density.c/93
  68 2988 0             DM_Vx    126.8573   27470.9837  subfind_density()/subfind_density.c/95
  68 2989 0             DM_Vy    126.8573   27597.8410  subfind_density()/subfind_density.c/96
  68 2990 0             DM_Vz    126.8573   27724.6983  subfind_density()/subfind_density.c/97
  68 2991 0    	  DensDataGet    124.8875   27849.5858  subfind_density()/subfind_density.c/161
  68 2992 0  	  DensDataResult     93.6656   27943.2515  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=68: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.7485 MB).
task 68: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 68
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  61 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  61 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  61 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  61 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  61 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  61 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  61 2973 0         FOF_PList    380.6075   18861.4596  fof_fof()/fof.c/249
  61 2974 0         FOF_GList    634.3458   19495.8054  fof_fof()/fof.c/337
  61 2975 0             Group     12.4823   19508.2877  fof_fof()/fof.c/380
  61 2976 0   DomainNodeIndex      0.3495   19508.6372  force_treeallocate()/forcetree.c/4800
  61 2977 0        Nodes_base   3737.6006   23246.2379  force_treeallocate()/forcetree.c/4803
  61 2978 0     Extnodes_base   3114.6672   26360.9051  force_treeallocate()/forcetree.c/4811
  61 2979 0          Nextnode    222.3050   26583.2101  force_treeallocate()/forcetree.c/4820
  61 2980 0            Father    221.9056   26805.1157  force_treeallocate()/forcetree.c/4827
  61 2981 0           Ngblist     63.4346   26868.5503  subfind_density()/subfind_density.c/78
  61 2982 0         Dist2list    126.8692   26995.4194  subfind_density()/subfind_density.c/79
  61 2983 0    DataIndexTable     21.8182   27017.2376  subfind_density()/subfind_density.c/87
  61 2984 0      DataNodeList     58.1818   27075.4194  subfind_density()/subfind_density.c/89
  61 2985 0              Left    126.8692   27202.2886  subfind_density()/subfind_density.c/91
  61 2986 0             Right    126.8692   27329.1577  subfind_density()/subfind_density.c/92
  61 2987 0              Todo     15.8587   27345.0164  subfind_density()/subfind_density.c/93
  61 2988 0             DM_Vx    126.8692   27471.8855  subfind_density()/subfind_density.c/95
  61 2989 0             DM_Vy    126.8692   27598.7547  subfind_density()/subfind_density.c/96
  61 2990 0             DM_Vz    126.8692   27725.6238  subfind_density()/subfind_density.c/97
  61 2991 0    	  DensDataGet    108.0308   27833.6547  subfind_density()/subfind_density.c/161
  61 2992 0  	  DensDataResult     81.0231   27914.6778  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=61: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.3222 MB).
task 61: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 61
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  47 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  47 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  47 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  47 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  47 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  47 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  47 2973 0         FOF_PList    380.4742   18861.3264  fof_fof()/fof.c/249
  47 2974 0         FOF_GList    634.1237   19495.4501  fof_fof()/fof.c/337
  47 2975 0             Group     11.7591   19507.2092  fof_fof()/fof.c/380
  47 2976 0   DomainNodeIndex      0.3495   19507.5587  force_treeallocate()/forcetree.c/4800
  47 2977 0        Nodes_base   3737.6006   23245.1593  force_treeallocate()/forcetree.c/4803
  47 2978 0     Extnodes_base   3114.6672   26359.8265  force_treeallocate()/forcetree.c/4811
  47 2979 0          Nextnode    222.3050   26582.1316  force_treeallocate()/forcetree.c/4820
  47 2980 0            Father    221.9056   26804.0372  force_treeallocate()/forcetree.c/4827
  47 2981 0           Ngblist     63.4124   26867.4495  subfind_density()/subfind_density.c/78
  47 2982 0         Dist2list    126.8247   26994.2743  subfind_density()/subfind_density.c/79
  47 2983 0    DataIndexTable     21.8182   27016.0924  subfind_density()/subfind_density.c/87
  80 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  80 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  80 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  80 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  80 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  80 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  80 2973 0         FOF_PList    380.2768   18861.1290  fof_fof()/fof.c/249
  80 2974 0         FOF_GList    633.7947   19494.9237  fof_fof()/fof.c/337
  80 2975 0             Group     10.3397   19505.2634  fof_fof()/fof.c/380
  80 2976 0   DomainNodeIndex      0.3495   19505.6129  force_treeallocate()/forcetree.c/4800
  80 2977 0        Nodes_base   3737.6006   23243.2135  force_treeallocate()/forcetree.c/4803
  47 2984 0      DataNodeList     58.1818   27074.2742  subfind_density()/subfind_density.c/89
  47 2985 0              Left    126.8247   27201.0990  subfind_density()/subfind_density.c/91
  47 2986 0             Right    126.8247   27327.9237  subfind_density()/subfind_density.c/92
  47 2987 0              Todo     15.8531   27343.7768  subfind_density()/subfind_density.c/93
  47 2988 0             DM_Vx    126.8247   27470.6015  subfind_density()/subfind_density.c/95
  47 2989 0             DM_Vy    126.8247   27597.4263  subfind_density()/subfind_density.c/96
  80 2978 0     Extnodes_base   3114.6672   26357.8807  force_treeallocate()/forcetree.c/4811
  80 2979 0          Nextnode    222.3050   26580.1858  force_treeallocate()/forcetree.c/4820
  80 2980 0            Father    221.9056   26802.0914  force_treeallocate()/forcetree.c/4827
  80 2981 0           Ngblist     63.3795   26865.4708  subfind_density()/subfind_density.c/78
  47 2990 0             DM_Vz    126.8247   27724.2510  subfind_density()/subfind_density.c/97
  47 2991 0    	  DensDataGet    133.4581   27857.7091  subfind_density()/subfind_density.c/161
  47 2992 0  	  DensDataResult    100.0936   27957.8026  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  80 2982 0         Dist2list    126.7589   26992.2298  subfind_density()/subfind_density.c/79
  80 2983 0    DataIndexTable     21.8182   27014.0480  subfind_density()/subfind_density.c/87
  80 2984 0      DataNodeList     58.1818   27072.2297  subfind_density()/subfind_density.c/89

Task=47: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.1974 MB).
task 47: endrun called with an error level of 812


  80 2985 0              Left    126.7589   27198.9887  subfind_density()/subfind_density.c/91
  80 2986 0             Right    126.7589   27325.7476  subfind_density()/subfind_density.c/92
  80 2987 0              Todo     15.8449   27341.5925  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 47
  80 2988 0             DM_Vx    126.7589   27468.3514  subfind_density()/subfind_density.c/95
  80 2989 0             DM_Vy    126.7589   27595.1104  subfind_density()/subfind_density.c/96
  80 2990 0             DM_Vz    126.7589   27721.8693  subfind_density()/subfind_density.c/97
  80 2991 0    	  DensDataGet    126.6590   27848.5283  subfind_density()/subfind_density.c/161
  80 2992 0  	  DensDataResult     94.9942   27943.5226  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=80: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.4774 MB).
task 80: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 80
task 80: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 80:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 80
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  46 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  46 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  46 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  46 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  46 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  46 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  81 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  81 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  81 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  81 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  81 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  81 2973 0         FOF_PList    380.6053   18861.4575  fof_fof()/fof.c/249
  81 2974 0         FOF_GList    634.3422   19495.7997  fof_fof()/fof.c/337
  81 2975 0             Group     12.0486   19507.8483  fof_fof()/fof.c/380
  81 2976 0   DomainNodeIndex      0.3495   19508.1978  force_treeallocate()/forcetree.c/4800
  81 2977 0        Nodes_base   3737.6006   23245.7984  force_treeallocate()/forcetree.c/4803
  81 2978 0     Extnodes_base   3114.6672   26360.4656  force_treeallocate()/forcetree.c/4811
  46 2973 0         FOF_PList    380.6180   18861.4701  fof_fof()/fof.c/249
  46 2974 0         FOF_GList    634.3633   19495.8334  fof_fof()/fof.c/337
  46 2975 0             Group     11.3544   19507.1879  fof_fof()/fof.c/380
  81 2979 0          Nextnode    222.3050   26582.7707  force_treeallocate()/forcetree.c/4820
  81 2980 0            Father    221.9056   26804.6763  force_treeallocate()/forcetree.c/4827
  81 2981 0           Ngblist     63.4342   26868.1105  subfind_density()/subfind_density.c/78
  81 2982 0         Dist2list    126.8684   26994.9789  subfind_density()/subfind_density.c/79
  46 2976 0   DomainNodeIndex      0.3495   19507.5374  force_treeallocate()/forcetree.c/4800
  46 2977 0        Nodes_base   3737.6006   23245.1380  force_treeallocate()/forcetree.c/4803
  46 2978 0     Extnodes_base   3114.6672   26359.8052  force_treeallocate()/forcetree.c/4811
  81 2983 0    DataIndexTable     21.8182   27016.7971  subfind_density()/subfind_density.c/87
  81 2984 0      DataNodeList     58.1818   27074.9789  subfind_density()/subfind_density.c/89
  81 2985 0              Left    126.8684   27201.8473  subfind_density()/subfind_density.c/91
  46 2979 0          Nextnode    222.3050   26582.1103  force_treeallocate()/forcetree.c/4820
  46 2980 0            Father    221.9056   26804.0159  force_treeallocate()/forcetree.c/4827
  81 2986 0             Right    126.8684   27328.7158  subfind_density()/subfind_density.c/92
  81 2987 0              Todo     15.8586   27344.5743  subfind_density()/subfind_density.c/93
  81 2988 0             DM_Vx    126.8684   27471.4428  subfind_density()/subfind_density.c/95
  81 2989 0             DM_Vy    126.8684   27598.3112  subfind_density()/subfind_density.c/96
  46 2981 0           Ngblist     63.4363   26867.4522  subfind_density()/subfind_density.c/78
  46 2982 0         Dist2list    126.8727   26994.3249  subfind_density()/subfind_density.c/79
  46 2983 0    DataIndexTable     21.8182   27016.1430  subfind_density()/subfind_density.c/87
  81 2990 0             DM_Vz    126.8684   27725.1796  subfind_density()/subfind_density.c/97
  81 2991 0    	  DensDataGet    116.5923   27841.7719  subfind_density()/subfind_density.c/161
  81 2992 0  	  DensDataResult     87.4442   27929.2161  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  46 2984 0      DataNodeList     58.1818   27074.3248  subfind_density()/subfind_density.c/89
  46 2985 0              Left    126.8727   27201.1975  subfind_density()/subfind_density.c/91
  46 2986 0             Right    126.8727   27328.0701  subfind_density()/subfind_density.c/92

Task=81: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.7839 MB).
task 81: endrun called with an error level of 812


  46 2987 0              Todo     15.8591   27343.9292  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 81
  46 2988 0             DM_Vx    126.8727   27470.8019  subfind_density()/subfind_density.c/95
  46 2989 0             DM_Vy    126.8727   27597.6745  subfind_density()/subfind_density.c/96
  46 2990 0             DM_Vz    126.8727   27724.5472  subfind_density()/subfind_density.c/97
  46 2991 0    	  DensDataGet    115.9713   27840.5185  subfind_density()/subfind_density.c/161
  46 2992 0  	  DensDataResult     86.9785   27927.4970  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=46: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.503 MB).
task 46: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 46
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 122 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 122 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 122 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 122 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 122 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 122 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 122 2973 0         FOF_PList    380.0035   18860.8557  fof_fof()/fof.c/249
 122 2974 0         FOF_GList    633.3392   19494.1948  fof_fof()/fof.c/337
 122 2975 0             Group     10.3397   19504.5345  fof_fof()/fof.c/380
 122 2976 0   DomainNodeIndex      0.3495   19504.8840  force_treeallocate()/forcetree.c/4800
 122 2977 0        Nodes_base   3737.6006   23242.4846  force_treeallocate()/forcetree.c/4803
 122 2978 0     Extnodes_base   3114.6672   26357.1519  force_treeallocate()/forcetree.c/4811
 122 2979 0          Nextnode    222.3050   26579.4569  force_treeallocate()/forcetree.c/4820
   5 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   5 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   5 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   5 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   5 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   5 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   5 2973 0         FOF_PList    380.4762   18861.3284  fof_fof()/fof.c/249
   5 2974 0         FOF_GList    634.1270   19495.4553  fof_fof()/fof.c/337
   5 2975 0             Group     12.5383   19507.9937  fof_fof()/fof.c/380
   5 2976 0   DomainNodeIndex      0.3495   19508.3432  force_treeallocate()/forcetree.c/4800
   5 2977 0        Nodes_base   3737.6006   23245.9438  force_treeallocate()/forcetree.c/4803
   5 2978 0     Extnodes_base   3114.6672   26360.6110  force_treeallocate()/forcetree.c/4811
   5 2979 0          Nextnode    222.3050   26582.9161  force_treeallocate()/forcetree.c/4820
   5 2980 0            Father    221.9056   26804.8216  force_treeallocate()/forcetree.c/4827
 122 2980 0            Father    221.9056   26801.3625  force_treeallocate()/forcetree.c/4827
 122 2981 0           Ngblist     63.3339   26864.6964  subfind_density()/subfind_density.c/78
 122 2982 0         Dist2list    126.6678   26991.3642  subfind_density()/subfind_density.c/79
   5 2981 0           Ngblist     63.4127   26868.2343  subfind_density()/subfind_density.c/78
   5 2982 0         Dist2list    126.8254   26995.0597  subfind_density()/subfind_density.c/79
   5 2983 0    DataIndexTable     21.8182   27016.8779  subfind_density()/subfind_density.c/87
   5 2984 0      DataNodeList     58.1818   27075.0597  subfind_density()/subfind_density.c/89
 122 2983 0    DataIndexTable     21.8182   27013.1824  subfind_density()/subfind_density.c/87
 122 2984 0      DataNodeList     58.1818   27071.3642  subfind_density()/subfind_density.c/89
 122 2985 0              Left    126.6678   27198.0320  subfind_density()/subfind_density.c/91
   5 2985 0              Left    126.8254   27201.8851  subfind_density()/subfind_density.c/91
   5 2986 0             Right    126.8254   27328.7105  subfind_density()/subfind_density.c/92
   5 2987 0              Todo     15.8532   27344.5637  subfind_density()/subfind_density.c/93
   5 2988 0             DM_Vx    126.8254   27471.3891  subfind_density()/subfind_density.c/95
 122 2986 0             Right    126.6678   27324.6999  subfind_density()/subfind_density.c/92
 122 2987 0              Todo     15.8335   27340.5333  subfind_density()/subfind_density.c/93
 122 2988 0             DM_Vx    126.6678   27467.2012  subfind_density()/subfind_density.c/95
   5 2989 0             DM_Vy    126.8254   27598.2145  subfind_density()/subfind_density.c/96
   5 2990 0             DM_Vz    126.8254   27725.0399  subfind_density()/subfind_density.c/97
   5 2991 0    	  DensDataGet    126.9631   27852.0029  subfind_density()/subfind_density.c/161
 122 2989 0             DM_Vy    126.6678   27593.8690  subfind_density()/subfind_density.c/96
 122 2990 0             DM_Vz    126.6678   27720.5368  subfind_density()/subfind_density.c/97
 122 2991 0    	  DensDataGet    120.5928   27841.1296  subfind_density()/subfind_density.c/161
   5 2992 0  	  DensDataResult     95.2223   27947.2252  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=5: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.7748 MB).
task 5: endrun called with an error level of 812


 122 2992 0  	  DensDataResult     90.4446   27931.5742  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 5

Task=122: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.4258 MB).
task 122: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 122
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  32 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  32 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  32 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  32 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  32 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  32 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  32 2973 0         FOF_PList    380.4939   18861.3460  fof_fof()/fof.c/249
  32 2974 0         FOF_GList    634.1564   19495.5024  fof_fof()/fof.c/337
  32 2975 0             Group     13.5842   19509.0867  fof_fof()/fof.c/380
  32 2976 0   DomainNodeIndex      0.3495   19509.4362  force_treeallocate()/forcetree.c/4800
  32 2977 0        Nodes_base   3737.6006   23247.0368  force_treeallocate()/forcetree.c/4803
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  95 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  95 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  95 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  95 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  95 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  95 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  95 2973 0         FOF_PList    380.3583   18861.2105  fof_fof()/fof.c/249
  95 2974 0         FOF_GList    633.9305   19495.1410  fof_fof()/fof.c/337
  95 2975 0             Group     10.7589   19505.8998  fof_fof()/fof.c/380
  95 2976 0   DomainNodeIndex      0.3495   19506.2494  force_treeallocate()/forcetree.c/4800
  95 2977 0        Nodes_base   3737.6006   23243.8500  force_treeallocate()/forcetree.c/4803
  95 2978 0     Extnodes_base   3114.6672   26358.5172  force_treeallocate()/forcetree.c/4811
  95 2979 0          Nextnode    222.3050   26580.8223  force_treeallocate()/forcetree.c/4820
  95 2980 0            Father    221.9056   26802.7278  force_treeallocate()/forcetree.c/4827
  95 2981 0           Ngblist     63.3931   26866.1209  subfind_density()/subfind_density.c/78
  95 2982 0         Dist2list    126.7861   26992.9070  subfind_density()/subfind_density.c/79
  95 2983 0    DataIndexTable     21.8182   27014.7252  subfind_density()/subfind_density.c/87
  87 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  87 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  87 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  87 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  87 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  87 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  87 2973 0         FOF_PList    380.2248   18861.0770  fof_fof()/fof.c/249
  87 2974 0         FOF_GList    633.7081   19494.7851  fof_fof()/fof.c/337
  87 2975 0             Group     12.2976   19507.0827  fof_fof()/fof.c/380
  87 2976 0   DomainNodeIndex      0.3495   19507.4322  force_treeallocate()/forcetree.c/4800
  87 2977 0        Nodes_base   3737.6006   23245.0329  force_treeallocate()/forcetree.c/4803
  87 2978 0     Extnodes_base   3114.6672   26359.7001  force_treeallocate()/forcetree.c/4811
  87 2979 0          Nextnode    222.3050   26582.0051  force_treeallocate()/forcetree.c/4820
  87 2980 0            Father    221.9056   26803.9107  force_treeallocate()/forcetree.c/4827
  87 2981 0           Ngblist     63.3708   26867.2815  subfind_density()/subfind_density.c/78
  32 2978 0     Extnodes_base   3114.6672   26361.7040  force_treeallocate()/forcetree.c/4811
  32 2979 0          Nextnode    222.3050   26584.0091  force_treeallocate()/forcetree.c/4820
  40 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  40 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  40 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  40 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  40 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  40 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  40 2973 0         FOF_PList    380.5206   18861.3728  fof_fof()/fof.c/249
  40 2974 0         FOF_GList    634.2010   19495.5738  fof_fof()/fof.c/337
  40 2975 0             Group     13.0187   19508.5925  fof_fof()/fof.c/380
  95 2984 0      DataNodeList     58.1818   27072.9070  subfind_density()/subfind_density.c/89
  95 2985 0              Left    126.7861   27199.6931  subfind_density()/subfind_density.c/91
  95 2986 0             Right    126.7861   27326.4792  subfind_density()/subfind_density.c/92
  95 2987 0              Todo     15.8483   27342.3274  subfind_density()/subfind_density.c/93
  95 2988 0             DM_Vx    126.7861   27469.1135  subfind_density()/subfind_density.c/95
  95 2989 0             DM_Vy    126.7861   27595.8996  subfind_density()/subfind_density.c/96
  95 2990 0             DM_Vz    126.7861   27722.6857  subfind_density()/subfind_density.c/97
  87 2982 0         Dist2list    126.7416   26994.0231  subfind_density()/subfind_density.c/79
  87 2983 0    DataIndexTable     21.8182   27015.8413  subfind_density()/subfind_density.c/87
  87 2984 0      DataNodeList     58.1818   27074.0231  subfind_density()/subfind_density.c/89
  87 2985 0              Left    126.7416   27200.7647  subfind_density()/subfind_density.c/91
  87 2986 0             Right    126.7416   27327.5063  subfind_density()/subfind_density.c/92
  32 2980 0            Father    221.9056   26805.9147  force_treeallocate()/forcetree.c/4827
  32 2981 0           Ngblist     63.4156   26869.3303  subfind_density()/subfind_density.c/78
  40 2976 0   DomainNodeIndex      0.3495   19508.9420  force_treeallocate()/forcetree.c/4800
  40 2977 0        Nodes_base   3737.6006   23246.5427  force_treeallocate()/forcetree.c/4803
  95 2991 0    	  DensDataGet    115.9017   27838.5875  subfind_density()/subfind_density.c/161
  95 2992 0  	  DensDataResult     86.9263   27925.5138  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=95: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.4862 MB).
task 95: endrun called with an error level of 812


  87 2987 0              Todo     15.8427   27343.3490  subfind_density()/subfind_density.c/93
  87 2988 0             DM_Vx    126.7416   27470.0906  subfind_density()/subfind_density.c/95
  87 2989 0             DM_Vy    126.7416   27596.8323  subfind_density()/subfind_density.c/96
  87 2990 0             DM_Vz    126.7416   27723.5739  subfind_density()/subfind_density.c/97
  32 2982 0         Dist2list    126.8313   26996.1616  subfind_density()/subfind_density.c/79
  32 2983 0    DataIndexTable     21.8182   27017.9798  subfind_density()/subfind_density.c/87
  32 2984 0      DataNodeList     58.1818   27076.1616  subfind_density()/subfind_density.c/89
  40 2978 0     Extnodes_base   3114.6672   26361.2099  force_treeallocate()/forcetree.c/4811
  40 2979 0          Nextnode    222.3050   26583.5149  force_treeallocate()/forcetree.c/4820
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 95
  87 2991 0    	  DensDataGet    139.1934   27862.7673  subfind_density()/subfind_density.c/161
  87 2992 0  	  DensDataResult    104.3951   27967.1624  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=87: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=32.8376 MB).
  32 2985 0              Left    126.8313   27202.9928  subfind_density()/subfind_density.c/91
  32 2986 0             Right    126.8313   27329.8241  subfind_density()/subfind_density.c/92
  40 2980 0            Father    221.9056   26805.4205  force_treeallocate()/forcetree.c/4827
  40 2981 0           Ngblist     63.4201   26868.8406  subfind_density()/subfind_density.c/78
  40 2982 0         Dist2list    126.8402   26995.6808  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
task 87: endrun called with an error level of 812


  32 2987 0              Todo     15.8539   27345.6780  subfind_density()/subfind_density.c/93
  40 2983 0    DataIndexTable     21.8182   27017.4990  subfind_density()/subfind_density.c/87
  40 2984 0      DataNodeList     58.1818   27075.6808  subfind_density()/subfind_density.c/89
  89 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  89 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  89 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  89 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  89 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  89 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  89 2973 0         FOF_PList    379.8170   18860.6692  fof_fof()/fof.c/249
  89 2974 0         FOF_GList    633.0283   19493.6975  fof_fof()/fof.c/337
  89 2975 0             Group     11.3036   19505.0011  fof_fof()/fof.c/380
  89 2976 0   DomainNodeIndex      0.3495   19505.3506  force_treeallocate()/forcetree.c/4800
  89 2977 0        Nodes_base   3737.6006   23242.9513  force_treeallocate()/forcetree.c/4803
  89 2978 0     Extnodes_base   3114.6672   26357.6185  force_treeallocate()/forcetree.c/4811
  89 2979 0          Nextnode    222.3050   26579.9235  force_treeallocate()/forcetree.c/4820
  89 2980 0            Father    221.9056   26801.8291  force_treeallocate()/forcetree.c/4827
  89 2981 0           Ngblist     63.3028   26865.1319  subfind_density()/subfind_density.c/78
  89 2982 0         Dist2list    126.6057   26991.7376  subfind_density()/subfind_density.c/79
  89 2983 0    DataIndexTable     21.8182   27013.5558  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 87
  32 2988 0             DM_Vx    126.8313   27472.5093  subfind_density()/subfind_density.c/95
  40 2985 0              Left    126.8402   27202.5210  subfind_density()/subfind_density.c/91
  40 2986 0             Right    126.8402   27329.3612  subfind_density()/subfind_density.c/92
  40 2987 0              Todo     15.8550   27345.2162  subfind_density()/subfind_density.c/93
  89 2984 0      DataNodeList     58.1818   27071.7376  subfind_density()/subfind_density.c/89
  89 2985 0              Left    126.6057   27198.3432  subfind_density()/subfind_density.c/91
  89 2986 0             Right    126.6057   27324.9489  subfind_density()/subfind_density.c/92
  89 2987 0              Todo     15.8257   27340.7746  subfind_density()/subfind_density.c/93
  89 2988 0             DM_Vx    126.6057   27467.3803  subfind_density()/subfind_density.c/95
  89 2989 0             DM_Vy    126.6057   27593.9859  subfind_density()/subfind_density.c/96
  32 2989 0             DM_Vy    126.8313   27599.3406  subfind_density()/subfind_density.c/96
  40 2988 0             DM_Vx    126.8402   27472.0564  subfind_density()/subfind_density.c/95
  40 2989 0             DM_Vy    126.8402   27598.8966  subfind_density()/subfind_density.c/96
  89 2990 0             DM_Vz    126.6057   27720.5916  subfind_density()/subfind_density.c/97
  89 2991 0    	  DensDataGet    113.4058   27833.9974  subfind_density()/subfind_density.c/161
  89 2992 0  	  DensDataResult     85.0544   27919.0518  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  32 2990 0             DM_Vz    126.8313   27726.1719  subfind_density()/subfind_density.c/97
  40 2990 0             DM_Vz    126.8402   27725.7368  subfind_density()/subfind_density.c/97

Task=89: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.9482 MB).
task 89: endrun called with an error level of 812


  32 2991 0    	  DensDataGet    113.7768   27839.9487  subfind_density()/subfind_density.c/161
  40 2991 0    	  DensDataGet    110.8652   27836.6021  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 89
  32 2992 0  	  DensDataResult     85.3326   27925.2813  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  40 2992 0  	  DensDataResult     83.1489   27919.7510  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=32: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.7187 MB).
task 32: endrun called with an error level of 812



Task=40: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.249 MB).
task 40: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 32
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 40
 An HDF5 error was detected, we'd better stop.
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 40MPI-process 32task 40: endrun called with an error level of 124


:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 40
    major: Error API
    minor: Can't get value
task 32: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 32
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  86 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  86 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  86 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  86 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  86 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  86 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  86 2973 0         FOF_PList    380.6343   18861.4865  fof_fof()/fof.c/249
  86 2974 0         FOF_GList    634.3905   19495.8770  fof_fof()/fof.c/337
  86 2975 0             Group     11.5599   19507.4368  fof_fof()/fof.c/380
  86 2976 0   DomainNodeIndex      0.3495   19507.7863  force_treeallocate()/forcetree.c/4800
  86 2977 0        Nodes_base   3737.6006   23245.3870  force_treeallocate()/forcetree.c/4803
  86 2978 0     Extnodes_base   3114.6672   26360.0542  force_treeallocate()/forcetree.c/4811
  86 2979 0          Nextnode    222.3050   26582.3592  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  86 2980 0            Father    221.9056   26804.2648  force_treeallocate()/forcetree.c/4827
  86 2981 0           Ngblist     63.4390   26867.7039  subfind_density()/subfind_density.c/78
  86 2982 0         Dist2list    126.8781   26994.5820  subfind_density()/subfind_density.c/79
   4 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   4 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   4 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   4 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   4 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   4 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   4 2973 0         FOF_PList    380.3806   18861.2328  fof_fof()/fof.c/249
   4 2974 0         FOF_GList    633.9677   19495.2004  fof_fof()/fof.c/337
   4 2975 0             Group     10.7568   19505.9572  fof_fof()/fof.c/380
  86 2983 0    DataIndexTable     21.8182   27016.4001  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   4 2976 0   DomainNodeIndex      0.3495   19506.3067  force_treeallocate()/forcetree.c/4800
   4 2977 0        Nodes_base   3737.6006   23243.9074  force_treeallocate()/forcetree.c/4803
  86 2984 0      DataNodeList     58.1818   27074.5819  subfind_density()/subfind_density.c/89
  82 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  82 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  82 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  82 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  82 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  82 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  82 2973 0         FOF_PList    380.5603   18861.4125  fof_fof()/fof.c/249
  82 2974 0         FOF_GList    634.2672   19495.6797  fof_fof()/fof.c/337
  82 2975 0             Group     11.7819   19507.4617  fof_fof()/fof.c/380
  82 2976 0   DomainNodeIndex      0.3495   19507.8112  force_treeallocate()/forcetree.c/4800
  82 2977 0        Nodes_base   3737.6006   23245.4118  force_treeallocate()/forcetree.c/4803
  82 2978 0     Extnodes_base   3114.6672   26360.0790  force_treeallocate()/forcetree.c/4811
   4 2978 0     Extnodes_base   3114.6672   26358.5746  force_treeallocate()/forcetree.c/4811
   4 2979 0          Nextnode    222.3050   26580.8796  force_treeallocate()/forcetree.c/4820
   4 2980 0            Father    221.9056   26802.7852  force_treeallocate()/forcetree.c/4827
   4 2981 0           Ngblist     63.3968   26866.1820  subfind_density()/subfind_density.c/78
  86 2985 0              Left    126.8781   27201.4600  subfind_density()/subfind_density.c/91
  82 2979 0          Nextnode    222.3050   26582.3841  force_treeallocate()/forcetree.c/4820
  82 2980 0            Father    221.9056   26804.2897  force_treeallocate()/forcetree.c/4827
  82 2981 0           Ngblist     63.4267   26867.7164  subfind_density()/subfind_density.c/78
   4 2982 0         Dist2list    126.7935   26992.9755  subfind_density()/subfind_density.c/79
   4 2983 0    DataIndexTable     21.8182   27014.7937  subfind_density()/subfind_density.c/87
   4 2984 0      DataNodeList     58.1818   27072.9755  subfind_density()/subfind_density.c/89
  86 2986 0             Right    126.8781   27328.3381  subfind_density()/subfind_density.c/92
  82 2982 0         Dist2list    126.8534   26994.5698  subfind_density()/subfind_density.c/79
  82 2983 0    DataIndexTable     21.8182   27016.3880  subfind_density()/subfind_density.c/87
  82 2984 0      DataNodeList     58.1818   27074.5698  subfind_density()/subfind_density.c/89
   4 2985 0              Left    126.7935   27199.7690  subfind_density()/subfind_density.c/91
   4 2986 0             Right    126.7935   27326.5625  subfind_density()/subfind_density.c/92
   4 2987 0              Todo     15.8492   27342.4117  subfind_density()/subfind_density.c/93
  86 2987 0              Todo     15.8598   27344.1979  subfind_density()/subfind_density.c/93
  82 2985 0              Left    126.8534   27201.4233  subfind_density()/subfind_density.c/91
  82 2986 0             Right    126.8534   27328.2767  subfind_density()/subfind_density.c/92
  82 2987 0              Todo     15.8567   27344.1334  subfind_density()/subfind_density.c/93
   4 2988 0             DM_Vx    126.7935   27469.2053  subfind_density()/subfind_density.c/95
   4 2989 0             DM_Vy    126.7935   27595.9988  subfind_density()/subfind_density.c/96
  86 2988 0             DM_Vx    126.8781   27471.0760  subfind_density()/subfind_density.c/95
  82 2988 0             DM_Vx    126.8534   27470.9868  subfind_density()/subfind_density.c/95
  82 2989 0             DM_Vy    126.8534   27597.8403  subfind_density()/subfind_density.c/96
  82 2990 0             DM_Vz    126.8534   27724.6937  subfind_density()/subfind_density.c/97
   4 2990 0             DM_Vz    126.7935   27722.7923  subfind_density()/subfind_density.c/97
  86 2989 0             DM_Vy    126.8781   27597.9541  subfind_density()/subfind_density.c/96
  82 2991 0    	  DensDataGet    150.0626   27874.7563  subfind_density()/subfind_density.c/161
   4 2991 0    	  DensDataGet    128.5639   27851.3562  subfind_density()/subfind_density.c/161
  86 2990 0             DM_Vz    126.8781   27724.8322  subfind_density()/subfind_density.c/97
  82 2992 0  	  DensDataResult    112.5470   27987.3033  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------   4 2992 0  	  DensDataResult     96.4229   27947.7792  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  86 2991 0    	  DensDataGet    109.7559   27834.5881  subfind_density()/subfind_density.c/161


Task=4: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.2208 MB).
task 4: endrun called with an error level of 812


  86 2992 0  	  DensDataResult     82.3169   27916.9049  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=82: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=12.6967 MB).
task 82: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 4

Task=86: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.0951 MB).
task 86: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 82
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 86
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   6 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   6 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   6 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   6 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   6 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   6 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   6 2973 0         FOF_PList    380.6092   18861.4614  fof_fof()/fof.c/249
   6 2974 0         FOF_GList    634.3486   19495.8100  fof_fof()/fof.c/337
   6 2975 0             Group     12.0787   19507.8887  fof_fof()/fof.c/380
   6 2976 0   DomainNodeIndex      0.3495   19508.2382  force_treeallocate()/forcetree.c/4800
   6 2977 0        Nodes_base   3737.6006   23245.8388  force_treeallocate()/forcetree.c/4803
   6 2978 0     Extnodes_base   3114.6672   26360.5060  force_treeallocate()/forcetree.c/4811
   6 2979 0          Nextnode    222.3050   26582.8111  force_treeallocate()/forcetree.c/4820
   6 2980 0            Father    221.9056   26804.7167  force_treeallocate()/forcetree.c/4827
   6 2981 0           Ngblist     63.4349   26868.1515  subfind_density()/subfind_density.c/78
   6 2982 0         Dist2list    126.8697   26995.0213  subfind_density()/subfind_density.c/79
   6 2983 0    DataIndexTable     21.8182   27016.8394  subfind_density()/subfind_density.c/87
   6 2984 0      DataNodeList     58.1818   27075.0212  subfind_density()/subfind_density.c/89
   6 2985 0              Left    126.8697   27201.8910  subfind_density()/subfind_density.c/91
   6 2986 0             Right    126.8697   27328.7607  subfind_density()/subfind_density.c/92
   6 2987 0              Todo     15.8587   27344.6194  subfind_density()/subfind_density.c/93
   6 2988 0             DM_Vx    126.8697   27471.4891  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   6 2989 0             DM_Vy    126.8697   27598.3589  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  36 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  36 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  36 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  36 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  36 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  36 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  36 2973 0         FOF_PList    380.6095   18861.4617  fof_fof()/fof.c/249
  36 2974 0         FOF_GList    634.3492   19495.8109  fof_fof()/fof.c/337
  36 2975 0             Group     11.3409   19507.1519  fof_fof()/fof.c/380
  36 2976 0   DomainNodeIndex      0.3495   19507.5014  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   6 2990 0             DM_Vz    126.8697   27725.2286  subfind_density()/subfind_density.c/97
  91 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  91 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  91 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  91 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  91 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  91 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  91 2973 0         FOF_PList    380.6278   18861.4799  fof_fof()/fof.c/249
  91 2974 0         FOF_GList    634.3796   19495.8596  fof_fof()/fof.c/337
  91 2975 0             Group     12.6379   19508.4975  fof_fof()/fof.c/380
  91 2976 0   DomainNodeIndex      0.3495   19508.8470  force_treeallocate()/forcetree.c/4800
  91 2977 0        Nodes_base   3737.6006   23246.4477  force_treeallocate()/forcetree.c/4803
  91 2978 0     Extnodes_base   3114.6672   26361.1149  force_treeallocate()/forcetree.c/4811
  91 2979 0          Nextnode    222.3050   26583.4199  force_treeallocate()/forcetree.c/4820
  91 2980 0            Father    221.9056   26805.3255  force_treeallocate()/forcetree.c/4827
  91 2981 0           Ngblist     63.4380   26868.7635  subfind_density()/subfind_density.c/78
  91 2982 0         Dist2list    126.8759   26995.6394  subfind_density()/subfind_density.c/79
  91 2983 0    DataIndexTable     21.8182   27017.4576  subfind_density()/subfind_density.c/87
  91 2984 0      DataNodeList     58.1818   27075.6394  subfind_density()/subfind_density.c/89
  35 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  35 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  35 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  35 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  35 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  35 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  35 2973 0         FOF_PList    380.3811   18861.2333  fof_fof()/fof.c/249
  35 2974 0         FOF_GList    633.9685   19495.2018  fof_fof()/fof.c/337
  35 2975 0             Group     11.0784   19506.2802  fof_fof()/fof.c/380
  35 2976 0   DomainNodeIndex      0.3495   19506.6297  force_treeallocate()/forcetree.c/4800
  35 2977 0        Nodes_base   3737.6006   23244.2304  force_treeallocate()/forcetree.c/4803
  35 2978 0     Extnodes_base   3114.6672   26358.8976  force_treeallocate()/forcetree.c/4811
  35 2979 0          Nextnode    222.3050   26581.2026  force_treeallocate()/forcetree.c/4820
  35 2980 0            Father    221.9056   26803.1082  force_treeallocate()/forcetree.c/4827
  36 2977 0        Nodes_base   3737.6006   23245.1020  force_treeallocate()/forcetree.c/4803
  36 2978 0     Extnodes_base   3114.6672   26359.7692  force_treeallocate()/forcetree.c/4811
  92 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  92 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  92 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  92 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  92 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  92 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  92 2973 0         FOF_PList    380.6337   18861.4858  fof_fof()/fof.c/249
  92 2974 0         FOF_GList    634.3895   19495.8753  fof_fof()/fof.c/337
  92 2975 0             Group     11.4271   19507.3024  fof_fof()/fof.c/380
  92 2976 0   DomainNodeIndex      0.3495   19507.6519  force_treeallocate()/forcetree.c/4800
  92 2977 0        Nodes_base   3737.6006   23245.2525  force_treeallocate()/forcetree.c/4803
   6 2991 0    	  DensDataGet    107.4631   27832.6917  subfind_density()/subfind_density.c/161
  91 2985 0              Left    126.8759   27202.5153  subfind_density()/subfind_density.c/91
  91 2986 0             Right    126.8759   27329.3912  subfind_density()/subfind_density.c/92
  91 2987 0              Todo     15.8595   27345.2507  subfind_density()/subfind_density.c/93
  91 2988 0             DM_Vx    126.8759   27472.1266  subfind_density()/subfind_density.c/95
  91 2989 0             DM_Vy    126.8759   27599.0025  subfind_density()/subfind_density.c/96
  91 2990 0             DM_Vz    126.8759   27725.8785  subfind_density()/subfind_density.c/97
  35 2981 0           Ngblist     63.3969   26866.5051  subfind_density()/subfind_density.c/78
  35 2982 0         Dist2list    126.7937   26993.2988  subfind_density()/subfind_density.c/79
  35 2983 0    DataIndexTable     21.8182   27015.1169  subfind_density()/subfind_density.c/87
  35 2984 0      DataNodeList     58.1818   27073.2987  subfind_density()/subfind_density.c/89
  36 2979 0          Nextnode    222.3050   26582.0743  force_treeallocate()/forcetree.c/4820
  36 2980 0            Father    221.9056   26803.9799  force_treeallocate()/forcetree.c/4827
  36 2981 0           Ngblist     63.4349   26867.4148  subfind_density()/subfind_density.c/78
  36 2982 0         Dist2list    126.8698   26994.2846  subfind_density()/subfind_density.c/79
  92 2978 0     Extnodes_base   3114.6672   26359.9197  force_treeallocate()/forcetree.c/4811
  92 2979 0          Nextnode    222.3050   26582.2248  force_treeallocate()/forcetree.c/4820
   6 2992 0  	  DensDataResult     80.5973   27913.2890  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=6: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.711 MB).
task 6: endrun called with an error level of 812


  91 2991 0    	  DensDataGet    137.8549   27863.7333  subfind_density()/subfind_density.c/161
  91 2992 0  	  DensDataResult    103.3911   27967.1245  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=91: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=32.8755 MB).
  35 2985 0              Left    126.7937   27200.0924  subfind_density()/subfind_density.c/91
  35 2986 0             Right    126.7937   27326.8861  subfind_density()/subfind_density.c/92
  35 2987 0              Todo     15.8492   27342.7353  subfind_density()/subfind_density.c/93
  35 2988 0             DM_Vx    126.7937   27469.5290  subfind_density()/subfind_density.c/95
  35 2989 0             DM_Vy    126.7937   27596.3227  subfind_density()/subfind_density.c/96
  36 2983 0    DataIndexTable     21.8182   27016.1028  subfind_density()/subfind_density.c/87
  36 2984 0      DataNodeList     58.1818   27074.2846  subfind_density()/subfind_density.c/89
  36 2985 0              Left    126.8698   27201.1544  subfind_density()/subfind_density.c/91
  92 2980 0            Father    221.9056   26804.1304  force_treeallocate()/forcetree.c/4827
  92 2981 0           Ngblist     63.4389   26867.5693  subfind_density()/subfind_density.c/78
  92 2982 0         Dist2list    126.8779   26994.4472  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 6
task 91: endrun called with an error level of 812


  35 2990 0             DM_Vz    126.7937   27723.1164  subfind_density()/subfind_density.c/97
  35 2991 0    	  DensDataGet    119.4103   27842.5268  subfind_density()/subfind_density.c/161
  35 2992 0  	  DensDataResult     89.5578   27932.0845  subfind_density()/subfind_density.c/201
  36 2986 0             Right    126.8698   27328.0243  subfind_density()/subfind_density.c/92
  36 2987 0              Todo     15.8587   27343.8830  subfind_density()/subfind_density.c/93
  36 2988 0             DM_Vx    126.8698   27470.7528  subfind_density()/subfind_density.c/95
  92 2983 0    DataIndexTable     21.8182   27016.2654  subfind_density()/subfind_density.c/87
  92 2984 0      DataNodeList     58.1818   27074.4472  subfind_density()/subfind_density.c/89
  92 2985 0              Left    126.8779   27201.3251  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 91
----------------------------------------------------------------------------------------

Task=35: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.9155 MB).
task 35: endrun called with an error level of 812


  36 2989 0             DM_Vy    126.8698   27597.6227  subfind_density()/subfind_density.c/96
  92 2986 0             Right    126.8779   27328.2030  subfind_density()/subfind_density.c/92
  92 2987 0              Todo     15.8597   27344.0627  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 35
  36 2990 0             DM_Vz    126.8698   27724.4925  subfind_density()/subfind_density.c/97
  92 2988 0             DM_Vx    126.8779   27470.9406  subfind_density()/subfind_density.c/95
  36 2991 0    	  DensDataGet    116.9322   27841.4247  subfind_density()/subfind_density.c/161
  92 2989 0             DM_Vy    126.8779   27597.8185  subfind_density()/subfind_density.c/96
  36 2992 0  	  DensDataResult     87.6991   27929.1239  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  92 2990 0             DM_Vz    126.8779   27724.6964  subfind_density()/subfind_density.c/97

Task=36: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.8761 MB).
task 36: endrun called with an error level of 812


  92 2991 0    	  DensDataGet    110.8247   27835.5211  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 36
  92 2992 0  	  DensDataResult     83.1185   27918.6396  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=92: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.3604 MB).
task 92: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  39 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  39 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  39 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  39 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  39 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  39 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  39 2973 0         FOF_PList    380.4025   18861.2547  fof_fof()/fof.c/249
  39 2974 0         FOF_GList    634.0042   19495.2590  fof_fof()/fof.c/337
  39 2975 0             Group     11.2061   19506.4650  fof_fof()/fof.c/380
  39 2976 0   DomainNodeIndex      0.3495   19506.8145  force_treeallocate()/forcetree.c/4800
  39 2977 0        Nodes_base   3737.6006   23244.4152  force_treeallocate()/forcetree.c/4803
  39 2978 0     Extnodes_base   3114.6672   26359.0824  force_treeallocate()/forcetree.c/4811
  39 2979 0          Nextnode    222.3050   26581.3874  force_treeallocate()/forcetree.c/4820
  39 2980 0            Father    221.9056   26803.2930  force_treeallocate()/forcetree.c/4827
  39 2981 0           Ngblist     63.4004   26866.6935  subfind_density()/subfind_density.c/78
  88 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  88 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  88 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  88 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  88 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  39 2982 0         Dist2list    126.8008   26993.4943  subfind_density()/subfind_density.c/79
  39 2983 0    DataIndexTable     21.8182   27015.3125  subfind_density()/subfind_density.c/87
  39 2984 0      DataNodeList     58.1818   27073.4943  subfind_density()/subfind_density.c/89
  39 2985 0              Left    126.8008   27200.2951  subfind_density()/subfind_density.c/91
  39 2986 0             Right    126.8008   27327.0960  subfind_density()/subfind_density.c/92
  88 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  88 2973 0         FOF_PList    380.4305   18861.2827  fof_fof()/fof.c/249
  88 2974 0         FOF_GList    634.0509   19495.3336  fof_fof()/fof.c/337
  39 2987 0              Todo     15.8501   27342.9461  subfind_density()/subfind_density.c/93
  39 2988 0             DM_Vx    126.8008   27469.7469  subfind_density()/subfind_density.c/95
  39 2989 0             DM_Vy    126.8008   27596.5478  subfind_density()/subfind_density.c/96
  88 2975 0             Group     11.3378   19506.6714  fof_fof()/fof.c/380
  88 2976 0   DomainNodeIndex      0.3495   19507.0209  force_treeallocate()/forcetree.c/4800
  88 2977 0        Nodes_base   3737.6006   23244.6216  force_treeallocate()/forcetree.c/4803
  39 2990 0             DM_Vz    126.8008   27723.3486  subfind_density()/subfind_density.c/97
  39 2991 0    	  DensDataGet    112.3113   27835.6600  subfind_density()/subfind_density.c/161
  39 2992 0  	  DensDataResult     84.2335   27919.8935  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  88 2978 0     Extnodes_base   3114.6672   26359.2888  force_treeallocate()/forcetree.c/4811
  88 2979 0          Nextnode    222.3050   26581.5938  force_treeallocate()/forcetree.c/4820
  88 2980 0            Father    221.9056   26803.4994  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=39: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.1065 MB).
task 39: endrun called with an error level of 812


  88 2981 0           Ngblist     63.4051   26866.9045  subfind_density()/subfind_density.c/78
  88 2982 0         Dist2list    126.8102   26993.7146  subfind_density()/subfind_density.c/79
  88 2983 0    DataIndexTable     21.8182   27015.5328  subfind_density()/subfind_density.c/87
  90 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  90 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  90 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  90 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  90 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  37 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  37 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  37 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  37 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  37 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  37 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  37 2973 0         FOF_PList    379.5782   18860.4303  fof_fof()/fof.c/249
  37 2974 0         FOF_GList    632.6303   19493.0606  fof_fof()/fof.c/337
  37 2975 0             Group     10.4258   19503.4864  fof_fof()/fof.c/380
  37 2976 0   DomainNodeIndex      0.3495   19503.8359  force_treeallocate()/forcetree.c/4800
  37 2977 0        Nodes_base   3737.6006   23241.4366  force_treeallocate()/forcetree.c/4803
  37 2978 0     Extnodes_base   3114.6672   26356.1038  force_treeallocate()/forcetree.c/4811
  37 2979 0          Nextnode    222.3050   26578.4088  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 39
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  88 2984 0      DataNodeList     58.1818   27073.7146  subfind_density()/subfind_density.c/89
  88 2985 0              Left    126.8102   27200.5248  subfind_density()/subfind_density.c/91
  90 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  90 2973 0         FOF_PList    380.1961   18861.0483  fof_fof()/fof.c/249
  37 2980 0            Father    221.9056   26800.3144  force_treeallocate()/forcetree.c/4827
  37 2981 0           Ngblist     63.2630   26863.5774  subfind_density()/subfind_density.c/78
  37 2982 0         Dist2list    126.5261   26990.1035  subfind_density()/subfind_density.c/79
  37 2983 0    DataIndexTable     21.8182   27011.9216  subfind_density()/subfind_density.c/87
  43 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  43 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  43 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  43 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  43 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  43 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  43 2973 0         FOF_PList    380.4517   18861.3039  fof_fof()/fof.c/249
  43 2974 0         FOF_GList    634.0862   19495.3901  fof_fof()/fof.c/337
  43 2975 0             Group     12.1669   19507.5569  fof_fof()/fof.c/380
  43 2976 0   DomainNodeIndex      0.3495   19507.9065  force_treeallocate()/forcetree.c/4800
  43 2977 0        Nodes_base   3737.6006   23245.5071  force_treeallocate()/forcetree.c/4803
  43 2978 0     Extnodes_base   3114.6672   26360.1743  force_treeallocate()/forcetree.c/4811
  43 2979 0          Nextnode    222.3050   26582.4794  force_treeallocate()/forcetree.c/4820
  43 2980 0            Father    221.9056   26804.3849  force_treeallocate()/forcetree.c/4827
  43 2981 0           Ngblist     63.4086   26867.7936  subfind_density()/subfind_density.c/78
  43 2982 0         Dist2list    126.8172   26994.6108  subfind_density()/subfind_density.c/79
  43 2983 0    DataIndexTable     21.8182   27016.4290  subfind_density()/subfind_density.c/87
  84 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  84 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  84 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  84 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  84 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  84 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  84 2973 0         FOF_PList    380.5988   18861.4510  fof_fof()/fof.c/249
  84 2974 0         FOF_GList    634.3314   19495.7824  fof_fof()/fof.c/337
  84 2975 0             Group     12.3173   19508.0997  fof_fof()/fof.c/380
  88 2986 0             Right    126.8102   27327.3350  subfind_density()/subfind_density.c/92
  90 2974 0         FOF_GList    633.6602   19494.7086  fof_fof()/fof.c/337
  90 2975 0             Group     10.7339   19505.4425  fof_fof()/fof.c/380
  90 2976 0   DomainNodeIndex      0.3495   19505.7920  force_treeallocate()/forcetree.c/4800
  37 2984 0      DataNodeList     58.1818   27070.1034  subfind_density()/subfind_density.c/89
  37 2985 0              Left    126.5261   27196.6295  subfind_density()/subfind_density.c/91
  37 2986 0             Right    126.5261   27323.1555  subfind_density()/subfind_density.c/92
  43 2984 0      DataNodeList     58.1818   27074.6108  subfind_density()/subfind_density.c/89
  43 2985 0              Left    126.8172   27201.4280  subfind_density()/subfind_density.c/91
  43 2986 0             Right    126.8172   27328.2452  subfind_density()/subfind_density.c/92
  43 2987 0              Todo     15.8522   27344.0974  subfind_density()/subfind_density.c/93
  43 2988 0             DM_Vx    126.8172   27470.9146  subfind_density()/subfind_density.c/95
  43 2989 0             DM_Vy    126.8172   27597.7319  subfind_density()/subfind_density.c/96
  43 2990 0             DM_Vz    126.8172   27724.5491  subfind_density()/subfind_density.c/97
  84 2976 0   DomainNodeIndex      0.3495   19508.4492  force_treeallocate()/forcetree.c/4800
  88 2987 0              Todo     15.8513   27343.1862  subfind_density()/subfind_density.c/93
  90 2977 0        Nodes_base   3737.6006   23243.3927  force_treeallocate()/forcetree.c/4803
  90 2978 0     Extnodes_base   3114.6672   26358.0599  force_treeallocate()/forcetree.c/4811
  90 2979 0          Nextnode    222.3050   26580.3649  force_treeallocate()/forcetree.c/4820
  37 2987 0              Todo     15.8158   27338.9713  subfind_density()/subfind_density.c/93
  37 2988 0             DM_Vx    126.5261   27465.4974  subfind_density()/subfind_density.c/95
  37 2989 0             DM_Vy    126.5261   27592.0234  subfind_density()/subfind_density.c/96
  43 2991 0    	  DensDataGet    116.9176   27841.4667  subfind_density()/subfind_density.c/161
  43 2992 0  	  DensDataResult     87.6882   27929.1549  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=43: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.8451 MB).
task 43: endrun called with an error level of 812


  84 2977 0        Nodes_base   3737.6006   23246.0499  force_treeallocate()/forcetree.c/4803
  84 2978 0     Extnodes_base   3114.6672   26360.7171  force_treeallocate()/forcetree.c/4811
  88 2988 0             DM_Vx    126.8102   27469.9964  subfind_density()/subfind_density.c/95
  90 2980 0            Father    221.9056   26802.2705  force_treeallocate()/forcetree.c/4827
  90 2981 0           Ngblist     63.3660   26865.6365  subfind_density()/subfind_density.c/78
  37 2990 0             DM_Vz    126.5261   27718.5495  subfind_density()/subfind_density.c/97
  37 2991 0    	  DensDataGet    113.0251   27831.5746  subfind_density()/subfind_density.c/161
  37 2992 0  	  DensDataResult     84.7688   27916.3434  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 43
  84 2979 0          Nextnode    222.3050   26583.0221  force_treeallocate()/forcetree.c/4820
  84 2980 0            Father    221.9056   26804.9277  force_treeallocate()/forcetree.c/4827
  84 2981 0           Ngblist     63.4331   26868.3609  subfind_density()/subfind_density.c/78
  88 2989 0             DM_Vy    126.8102   27596.8066  subfind_density()/subfind_density.c/96
  90 2982 0         Dist2list    126.7320   26992.3686  subfind_density()/subfind_density.c/79
  90 2983 0    DataIndexTable     21.8182   27014.1868  subfind_density()/subfind_density.c/87

Task=37: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.6566 MB).
task 37: endrun called with an error level of 812


  84 2982 0         Dist2list    126.8663   26995.2271  subfind_density()/subfind_density.c/79
  84 2983 0    DataIndexTable     21.8182   27017.0453  subfind_density()/subfind_density.c/87
  88 2990 0             DM_Vz    126.8102   27723.6168  subfind_density()/subfind_density.c/97
  90 2984 0      DataNodeList     58.1818   27072.3685  subfind_density()/subfind_density.c/89
  90 2985 0              Left    126.7320   27199.1006  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 37
  84 2984 0      DataNodeList     58.1818   27075.2271  subfind_density()/subfind_density.c/89
  88 2991 0    	  DensDataGet    109.2125   27832.8292  subfind_density()/subfind_density.c/161
  90 2986 0             Right    126.7320   27325.8326  subfind_density()/subfind_density.c/92
  84 2985 0              Left    126.8663   27202.0934  subfind_density()/subfind_density.c/91
  88 2992 0  	  DensDataResult     81.9093   27914.7386  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  90 2987 0              Todo     15.8415   27341.6741  subfind_density()/subfind_density.c/93
  84 2986 0             Right    126.8663   27328.9597  subfind_density()/subfind_density.c/92

Task=88: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.2614 MB).
task 88: endrun called with an error level of 812


  90 2988 0             DM_Vx    126.7320   27468.4062  subfind_density()/subfind_density.c/95
  84 2987 0              Todo     15.8583   27344.8180  subfind_density()/subfind_density.c/93
  84 2988 0             DM_Vx    126.8663   27471.6842  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 88
  90 2989 0             DM_Vy    126.7320   27595.1382  subfind_density()/subfind_density.c/96
  84 2989 0             DM_Vy    126.8663   27598.5505  subfind_density()/subfind_density.c/96
 An HDF5 error was detected, we'd better stop.
  90 2990 0             DM_Vz    126.7320   27721.8703  subfind_density()/subfind_density.c/97
  84 2990 0             DM_Vz    126.8663   27725.4168  subfind_density()/subfind_density.c/97
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 88:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
  90 2991 0    	  DensDataGet    136.1561   27858.0264  subfind_density()/subfind_density.c/161
  84 2991 0    	  DensDataGet    123.2561   27848.6729  subfind_density()/subfind_density.c/161
task 88: endrun called with an error level of 124


  90 2992 0  	  DensDataResult    102.1171   27960.1435  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  84 2992 0  	  DensDataResult     92.4421   27941.1150  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 88


Task=84: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.885 MB).
task 84: endrun called with an error level of 812



Task=90: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=39.8565 MB).
task 90: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 84
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 90
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  96 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  96 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  96 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  96 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  96 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  96 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  96 2973 0         FOF_PList    380.5331   18861.3853  fof_fof()/fof.c/249
  96 2974 0         FOF_GList    634.2218   19495.6071  fof_fof()/fof.c/337
  96 2975 0             Group     10.3397   19505.9467  fof_fof()/fof.c/380
  96 2976 0   DomainNodeIndex      0.3495   19506.2962  force_treeallocate()/forcetree.c/4800
  96 2977 0        Nodes_base   3737.6006   23243.8969  force_treeallocate()/forcetree.c/4803
  96 2978 0     Extnodes_base   3114.6672   26358.5641  force_treeallocate()/forcetree.c/4811
  96 2979 0          Nextnode    222.3050   26580.8691  force_treeallocate()/forcetree.c/4820
  96 2980 0            Father    221.9056   26802.7747  force_treeallocate()/forcetree.c/4827
  96 2981 0           Ngblist     63.4222   26866.1969  subfind_density()/subfind_density.c/78
  96 2982 0         Dist2list    126.8444   26993.0413  subfind_density()/subfind_density.c/79
  96 2983 0    DataIndexTable     21.8182   27014.8594  subfind_density()/subfind_density.c/87
  96 2984 0      DataNodeList     58.1818   27073.0412  subfind_density()/subfind_density.c/89
  96 2985 0              Left    126.8444   27199.8856  subfind_density()/subfind_density.c/91
  96 2986 0             Right    126.8444   27326.7299  subfind_density()/subfind_density.c/92
  96 2987 0              Todo     15.8555   27342.5855  subfind_density()/subfind_density.c/93
  96 2988 0             DM_Vx    126.8444   27469.4298  subfind_density()/subfind_density.c/95
  96 2989 0             DM_Vy    126.8444   27596.2742  subfind_density()/subfind_density.c/96
  96 2990 0             DM_Vz    126.8444   27723.1186  subfind_density()/subfind_density.c/97
  96 2991 0    	  DensDataGet    123.9109   27847.0295  subfind_density()/subfind_density.c/161
  96 2992 0  	  DensDataResult     92.9332   27939.9627  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=96: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.0373 MB).
task 96: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 96
task 96: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 96:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 120 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 120 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 120 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 120 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 120 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 120 2973 0         FOF_PList    380.6313   18861.4835  fof_fof()/fof.c/249
 120 2974 0         FOF_GList    634.3855   19495.8690  fof_fof()/fof.c/337
 120 2975 0             Group     12.2125   19508.0815  fof_fof()/fof.c/380
 120 2976 0   DomainNodeIndex      0.3495   19508.4311  force_treeallocate()/forcetree.c/4800
 120 2977 0        Nodes_base   3737.6006   23246.0317  force_treeallocate()/forcetree.c/4803
   7 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   7 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   7 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   7 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   7 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   7 2973 0         FOF_PList    380.6311   18861.4833  fof_fof()/fof.c/249
   7 2974 0         FOF_GList    634.3852   19495.8685  fof_fof()/fof.c/337
   7 2975 0             Group     11.9500   19507.8185  fof_fof()/fof.c/380
   7 2976 0   DomainNodeIndex      0.3495   19508.1680  force_treeallocate()/forcetree.c/4800
   7 2977 0        Nodes_base   3737.6006   23245.7686  force_treeallocate()/forcetree.c/4803
   7 2978 0     Extnodes_base   3114.6672   26360.4359  force_treeallocate()/forcetree.c/4811
   7 2979 0          Nextnode    222.3050   26582.7409  force_treeallocate()/forcetree.c/4820
   7 2980 0            Father    221.9056   26804.6465  force_treeallocate()/forcetree.c/4827
   7 2981 0           Ngblist     63.4385   26868.0850  subfind_density()/subfind_density.c/78
   7 2982 0         Dist2list    126.8770   26994.9620  subfind_density()/subfind_density.c/79
   7 2983 0    DataIndexTable     21.8182   27016.7802  subfind_density()/subfind_density.c/87
 120 2978 0     Extnodes_base   3114.6672   26360.6989  force_treeallocate()/forcetree.c/4811
   7 2984 0      DataNodeList     58.1818   27074.9620  subfind_density()/subfind_density.c/89
   7 2985 0              Left    126.8770   27201.8390  subfind_density()/subfind_density.c/91
   7 2986 0             Right    126.8770   27328.7161  subfind_density()/subfind_density.c/92
   7 2987 0              Todo     15.8596   27344.5757  subfind_density()/subfind_density.c/93
   7 2988 0             DM_Vx    126.8770   27471.4528  subfind_density()/subfind_density.c/95
 120 2979 0          Nextnode    222.3050   26583.0040  force_treeallocate()/forcetree.c/4820
 120 2980 0            Father    221.9056   26804.9095  force_treeallocate()/forcetree.c/4827
   7 2989 0             DM_Vy    126.8770   27598.3298  subfind_density()/subfind_density.c/96
   7 2990 0             DM_Vz    126.8770   27725.2068  subfind_density()/subfind_density.c/97
   7 2991 0    	  DensDataGet    138.0082   27863.2151  subfind_density()/subfind_density.c/161
   7 2992 0  	  DensDataResult    103.5062   27966.7212  subfind_density()/subfind_density.c/201
 120 2981 0           Ngblist     63.4386   26868.3481  subfind_density()/subfind_density.c/78
 120 2982 0         Dist2list    126.8771   26995.2252  subfind_density()/subfind_density.c/79
 120 2983 0    DataIndexTable     21.8182   27017.0434  subfind_density()/subfind_density.c/87
----------------------------------------------------------------------------------------

Task=7: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=33.2788 MB).
task 7: endrun called with an error level of 812


 120 2984 0      DataNodeList     58.1818   27075.2252  subfind_density()/subfind_density.c/89
 120 2985 0              Left    126.8771   27202.1023  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 7
 120 2986 0             Right    126.8771   27328.9794  subfind_density()/subfind_density.c/92
 120 2987 0              Todo     15.8596   27344.8390  subfind_density()/subfind_density.c/93
 120 2988 0             DM_Vx    126.8771   27471.7161  subfind_density()/subfind_density.c/95
 120 2989 0             DM_Vy    126.8771   27598.5932  subfind_density()/subfind_density.c/96
 120 2990 0             DM_Vz    126.8771   27725.4703  subfind_density()/subfind_density.c/97
 120 2991 0    	  DensDataGet    130.3477   27855.8181  subfind_density()/subfind_density.c/161
 120 2992 0  	  DensDataResult     97.7608   27953.5788  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=120: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.4212 MB).
task 120: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 120
task 120: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 120:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 120
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  30 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  30 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  30 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  30 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  30 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  30 2973 0         FOF_PList    380.5455   18861.3977  fof_fof()/fof.c/249
  30 2974 0         FOF_GList    634.2426   19495.6403  fof_fof()/fof.c/337
  30 2975 0             Group     11.8525   19507.4927  fof_fof()/fof.c/380
  30 2976 0   DomainNodeIndex      0.3495   19507.8423  force_treeallocate()/forcetree.c/4800
  30 2977 0        Nodes_base   3737.6006   23245.4429  force_treeallocate()/forcetree.c/4803
  30 2978 0     Extnodes_base   3114.6672   26360.1101  force_treeallocate()/forcetree.c/4811
  30 2979 0          Nextnode    222.3050   26582.4151  force_treeallocate()/forcetree.c/4820
  30 2980 0            Father    221.9056   26804.3207  force_treeallocate()/forcetree.c/4827
  30 2981 0           Ngblist     63.4243   26867.7450  subfind_density()/subfind_density.c/78
  30 2982 0         Dist2list    126.8485   26994.5935  subfind_density()/subfind_density.c/79
  30 2983 0    DataIndexTable     21.8182   27016.4117  subfind_density()/subfind_density.c/87
  30 2984 0      DataNodeList     58.1818   27074.5935  subfind_density()/subfind_density.c/89
  30 2985 0              Left    126.8485   27201.4420  subfind_density()/subfind_density.c/91
  30 2986 0             Right    126.8485   27328.2905  subfind_density()/subfind_density.c/92
  30 2987 0              Todo     15.8561   27344.1466  subfind_density()/subfind_density.c/93
  30 2988 0             DM_Vx    126.8485   27470.9951  subfind_density()/subfind_density.c/95
  30 2989 0             DM_Vy    126.8485   27597.8436  subfind_density()/subfind_density.c/96
  30 2990 0             DM_Vz    126.8485   27724.6921  subfind_density()/subfind_density.c/97
  30 2991 0    	  DensDataGet    132.2714   27856.9634  subfind_density()/subfind_density.c/161
  30 2992 0  	  DensDataResult     99.2035   27956.1670  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=30: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=43.833 MB).
task 30: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 30
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   1 2973 0         FOF_PList    380.6204   18861.4725  fof_fof()/fof.c/249
   1 2974 0         FOF_GList    634.3673   19495.8398  fof_fof()/fof.c/337
   1 2975 0             Group     11.0359   19506.8757  fof_fof()/fof.c/380
   1 2976 0   DomainNodeIndex      0.3495   19507.2253  force_treeallocate()/forcetree.c/4800
   1 2977 0        Nodes_base   3737.6006   23244.8259  force_treeallocate()/forcetree.c/4803
   1 2978 0     Extnodes_base   3114.6672   26359.4931  force_treeallocate()/forcetree.c/4811
   1 2979 0          Nextnode    222.3050   26581.7981  force_treeallocate()/forcetree.c/4820
 126 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 126 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 126 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 126 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 126 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 126 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 126 2973 0         FOF_PList    380.4480   18861.3002  fof_fof()/fof.c/249
 126 2974 0         FOF_GList    634.0800   19495.3802  fof_fof()/fof.c/337
 126 2975 0             Group     10.3397   19505.7199  fof_fof()/fof.c/380
 126 2976 0   DomainNodeIndex      0.3495   19506.0694  force_treeallocate()/forcetree.c/4800
   1 2980 0            Father    221.9056   26803.7037  force_treeallocate()/forcetree.c/4827
   1 2981 0           Ngblist     63.4367   26867.1405  subfind_density()/subfind_density.c/78
 126 2977 0        Nodes_base   3737.6006   23243.6701  force_treeallocate()/forcetree.c/4803
 126 2978 0     Extnodes_base   3114.6672   26358.3373  force_treeallocate()/forcetree.c/4811
 126 2979 0          Nextnode    222.3050   26580.6423  force_treeallocate()/forcetree.c/4820
   1 2982 0         Dist2list    126.8735   26994.0139  subfind_density()/subfind_density.c/79
 126 2980 0            Father    221.9056   26802.5479  force_treeallocate()/forcetree.c/4827
 126 2981 0           Ngblist     63.4080   26865.9559  subfind_density()/subfind_density.c/78
 126 2982 0         Dist2list    126.8160   26992.7719  subfind_density()/subfind_density.c/79
   1 2983 0    DataIndexTable     21.8182   27015.8321  subfind_density()/subfind_density.c/87
 126 2983 0    DataIndexTable     21.8182   27014.5901  subfind_density()/subfind_density.c/87
 126 2984 0      DataNodeList     58.1818   27072.7719  subfind_density()/subfind_density.c/89
 126 2985 0              Left    126.8160   27199.5879  subfind_density()/subfind_density.c/91
   1 2984 0      DataNodeList     58.1818   27074.0139  subfind_density()/subfind_density.c/89
 126 2986 0             Right    126.8160   27326.4039  subfind_density()/subfind_density.c/92
 126 2987 0              Todo     15.8520   27342.2559  subfind_density()/subfind_density.c/93
 126 2988 0             DM_Vx    126.8160   27469.0719  subfind_density()/subfind_density.c/95
   1 2985 0              Left    126.8735   27200.8873  subfind_density()/subfind_density.c/91
 126 2989 0             DM_Vy    126.8160   27595.8879  subfind_density()/subfind_density.c/96
 126 2990 0             DM_Vz    126.8160   27722.7039  subfind_density()/subfind_density.c/97
   1 2986 0             Right    126.8735   27327.7608  subfind_density()/subfind_density.c/92
 126 2991 0    	  DensDataGet    123.1522   27845.8562  subfind_density()/subfind_density.c/161
 126 2992 0  	  DensDataResult     92.3642   27938.2203  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   1 2987 0              Todo     15.8592   27343.6200  subfind_density()/subfind_density.c/93

Task=126: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.7797 MB).
task 126: endrun called with an error level of 812


   1 2988 0             DM_Vx    126.8735   27470.4934  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 126
   1 2989 0             DM_Vy    126.8735   27597.3669  subfind_density()/subfind_density.c/96
   1 2990 0             DM_Vz    126.8735   27724.2404  subfind_density()/subfind_density.c/97
   1 2991 0    	  DensDataGet    123.7817   27848.0220  subfind_density()/subfind_density.c/161
   1 2992 0  	  DensDataResult     92.8363   27940.8583  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=1: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.1417 MB).
task 1: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 1
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  17 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  17 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  17 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  17 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  17 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  17 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  17 2973 0         FOF_PList    380.5231   18861.3753  fof_fof()/fof.c/249
  17 2974 0         FOF_GList    634.2052   19495.5806  fof_fof()/fof.c/337
  17 2975 0             Group     10.5762   19506.1568  fof_fof()/fof.c/380
  17 2976 0   DomainNodeIndex      0.3495   19506.5063  force_treeallocate()/forcetree.c/4800
  17 2977 0        Nodes_base   3737.6006   23244.1070  force_treeallocate()/forcetree.c/4803
  17 2978 0     Extnodes_base   3114.6672   26358.7742  force_treeallocate()/forcetree.c/4811
  17 2979 0          Nextnode    222.3050   26581.0792  force_treeallocate()/forcetree.c/4820
  17 2980 0            Father    221.9056   26802.9848  force_treeallocate()/forcetree.c/4827
  17 2981 0           Ngblist     63.4205   26866.4053  subfind_density()/subfind_density.c/78
  17 2982 0         Dist2list    126.8410   26993.2464  subfind_density()/subfind_density.c/79
  17 2983 0    DataIndexTable     21.8182   27015.0645  subfind_density()/subfind_density.c/87
  17 2984 0      DataNodeList     58.1818   27073.2463  subfind_density()/subfind_density.c/89
  17 2985 0              Left    126.8410   27200.0874  subfind_density()/subfind_density.c/91
  17 2986 0             Right    126.8410   27326.9284  subfind_density()/subfind_density.c/92
  17 2987 0              Todo     15.8551   27342.7836  subfind_density()/subfind_density.c/93
  17 2988 0             DM_Vx    126.8410   27469.6246  subfind_density()/subfind_density.c/95
  17 2989 0             DM_Vy    126.8410   27596.4657  subfind_density()/subfind_density.c/96
  17 2990 0             DM_Vz    126.8410   27723.3067  subfind_density()/subfind_density.c/97
  17 2991 0    	  DensDataGet    108.7869   27832.0936  subfind_density()/subfind_density.c/161
  17 2992 0  	  DensDataResult     81.5902   27913.6838  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=17: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.3162 MB).
task 17: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 17
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 119 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 119 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 119 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 119 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 119 2973 0         FOF_PList    380.4148   18861.2669  fof_fof()/fof.c/249
 119 2974 0         FOF_GList    634.0246   19495.2916  fof_fof()/fof.c/337
 119 2975 0             Group     11.0867   19506.3783  fof_fof()/fof.c/380
 119 2976 0   DomainNodeIndex      0.3495   19506.7278  force_treeallocate()/forcetree.c/4800
 119 2977 0        Nodes_base   3737.6006   23244.3285  force_treeallocate()/forcetree.c/4803
 119 2978 0     Extnodes_base   3114.6672   26358.9957  force_treeallocate()/forcetree.c/4811
 119 2979 0          Nextnode    222.3050   26581.3007  force_treeallocate()/forcetree.c/4820
 119 2980 0            Father    221.9056   26803.2063  force_treeallocate()/forcetree.c/4827
 119 2981 0           Ngblist     63.4025   26866.6088  subfind_density()/subfind_density.c/78
 119 2982 0         Dist2list    126.8049   26993.4137  subfind_density()/subfind_density.c/79
 119 2983 0    DataIndexTable     21.8182   27015.2319  subfind_density()/subfind_density.c/87
 119 2984 0      DataNodeList     58.1818   27073.4137  subfind_density()/subfind_density.c/89
 119 2985 0              Left    126.8049   27200.2186  subfind_density()/subfind_density.c/91
 119 2986 0             Right    126.8049   27327.0235  subfind_density()/subfind_density.c/92
 119 2987 0              Todo     15.8506   27342.8741  subfind_density()/subfind_density.c/93
 119 2988 0             DM_Vx    126.8049   27469.6790  subfind_density()/subfind_density.c/95
 119 2989 0             DM_Vy    126.8049   27596.4840  subfind_density()/subfind_density.c/96
 119 2990 0             DM_Vz    126.8049   27723.2889  subfind_density()/subfind_density.c/97
 119 2991 0    	  DensDataGet    135.1105   27858.3994  subfind_density()/subfind_density.c/161
 119 2992 0  	  DensDataResult    101.3329   27959.7323  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=119: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=40.2677 MB).
task 119: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 119
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2965 0     ProcessedFlag     55.4764      57.4321  allocate_memory()/allocate.c/52
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2966 0  NextActiveParticle    221.9056     279.3377  allocate_memory()/allocate.c/55
  25 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  25 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  25 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  25 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  25 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  25 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  25 2973 0         FOF_PList    380.6073   18861.4595  fof_fof()/fof.c/249
  25 2974 0         FOF_GList    634.3455   19495.8050  fof_fof()/fof.c/337
  25 2975 0             Group     11.2745   19507.0795  fof_fof()/fof.c/380
  25 2976 0   DomainNodeIndex      0.3495   19507.4290  force_treeallocate()/forcetree.c/4800
  25 2977 0        Nodes_base   3737.6006   23245.0297  force_treeallocate()/forcetree.c/4803
  25 2978 0     Extnodes_base   3114.6672   26359.6969  force_treeallocate()/forcetree.c/4811
  25 2979 0          Nextnode    222.3050   26582.0019  force_treeallocate()/forcetree.c/4820
 102 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 102 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 102 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 102 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 102 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 102 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 102 2973 0         FOF_PList    380.0747   18860.9269  fof_fof()/fof.c/249
 102 2974 0         FOF_GList    633.4579   19494.3848  fof_fof()/fof.c/337
 102 2975 0             Group     11.5557   19505.9405  fof_fof()/fof.c/380
 102 2976 0   DomainNodeIndex      0.3495   19506.2900  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2967 0     NextInTimeBin    221.9056     501.2433  allocate_memory()/allocate.c/58
  25 2980 0            Father    221.9056   26803.9075  force_treeallocate()/forcetree.c/4827
  25 2981 0           Ngblist     63.4346   26867.3421  subfind_density()/subfind_density.c/78
  25 2982 0         Dist2list    126.8691   26994.2112  subfind_density()/subfind_density.c/79
 102 2977 0        Nodes_base   3737.6006   23243.8907  force_treeallocate()/forcetree.c/4803
 102 2978 0     Extnodes_base   3114.6672   26358.5579  force_treeallocate()/forcetree.c/4811
 127 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 127 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 127 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 127 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 127 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 127 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 127 2973 0         FOF_PList    380.3289   18861.1811  fof_fof()/fof.c/249
 127 2974 0         FOF_GList    633.8815   19495.0626  fof_fof()/fof.c/337
 127 2975 0             Group     11.9438   19507.0064  fof_fof()/fof.c/380
 127 2976 0   DomainNodeIndex      0.3495   19507.3559  force_treeallocate()/forcetree.c/4800
 127 2977 0        Nodes_base   3737.6006   23244.9566  force_treeallocate()/forcetree.c/4803
   0 2968 0     PrevInTimeBin    221.9056     723.1489  allocate_memory()/allocate.c/61
  25 2983 0    DataIndexTable     21.8182   27016.0294  subfind_density()/subfind_density.c/87
  25 2984 0      DataNodeList     58.1818   27074.2112  subfind_density()/subfind_density.c/89
  25 2985 0              Left    126.8691   27201.0803  subfind_density()/subfind_density.c/91
 102 2979 0          Nextnode    222.3050   26580.8629  force_treeallocate()/forcetree.c/4820
 102 2980 0            Father    221.9056   26802.7685  force_treeallocate()/forcetree.c/4827
 127 2978 0     Extnodes_base   3114.6672   26359.6238  force_treeallocate()/forcetree.c/4811
 127 2979 0          Nextnode    222.3050   26581.9288  force_treeallocate()/forcetree.c/4820
 127 2980 0            Father    221.9056   26803.8344  force_treeallocate()/forcetree.c/4827
 127 2981 0           Ngblist     63.3882   26867.2226  subfind_density()/subfind_density.c/78
 127 2982 0         Dist2list    126.7763   26993.9989  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2969 0                 P  17752.4469   18475.5958  allocate_memory()/allocate.c/67
  25 2986 0             Right    126.8691   27327.9494  subfind_density()/subfind_density.c/92
  25 2987 0              Todo     15.8586   27343.8080  subfind_density()/subfind_density.c/93
  25 2988 0             DM_Vx    126.8691   27470.6771  subfind_density()/subfind_density.c/95
 102 2981 0           Ngblist     63.3458   26866.1143  subfind_density()/subfind_density.c/78
 102 2982 0         Dist2list    126.6916   26992.8058  subfind_density()/subfind_density.c/79
 102 2983 0    DataIndexTable     21.8182   27014.6240  subfind_density()/subfind_density.c/87
 127 2983 0    DataIndexTable     21.8182   27015.8170  subfind_density()/subfind_density.c/87
 127 2984 0      DataNodeList     58.1818   27073.9988  subfind_density()/subfind_density.c/89
 127 2985 0              Left    126.7763   27200.7751  subfind_density()/subfind_density.c/91
 125 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 125 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 125 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 125 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 125 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 125 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 125 2973 0         FOF_PList    380.2418   18861.0940  fof_fof()/fof.c/249
 125 2974 0         FOF_GList    633.7363   19494.8303  fof_fof()/fof.c/337
 125 2975 0             Group     12.2530   19507.0833  fof_fof()/fof.c/380
 125 2976 0   DomainNodeIndex      0.3495   19507.4328  force_treeallocate()/forcetree.c/4800
 125 2977 0        Nodes_base   3737.6006   23245.0335  force_treeallocate()/forcetree.c/4803
 125 2978 0     Extnodes_base   3114.6672   26359.7007  force_treeallocate()/forcetree.c/4811
 125 2979 0          Nextnode    222.3050   26582.0057  force_treeallocate()/forcetree.c/4820
 125 2980 0            Father    221.9056   26803.9113  force_treeallocate()/forcetree.c/4827
 125 2981 0           Ngblist     63.3736   26867.2849  subfind_density()/subfind_density.c/78
 125 2982 0         Dist2list    126.7473   26994.0322  subfind_density()/subfind_density.c/79
 125 2983 0    DataIndexTable     21.8182   27015.8504  subfind_density()/subfind_density.c/87
 125 2984 0      DataNodeList     58.1818   27074.0322  subfind_density()/subfind_density.c/89
   0 2972 0          TopNodes      5.1928   18480.8511  domain_allocate()/domain.c/570
  25 2989 0             DM_Vy    126.8691   27597.5462  subfind_density()/subfind_density.c/96
  25 2990 0             DM_Vz    126.8691   27724.4153  subfind_density()/subfind_density.c/97
 102 2984 0      DataNodeList     58.1818   27072.8058  subfind_density()/subfind_density.c/89
 102 2985 0              Left    126.6916   27199.4974  subfind_density()/subfind_density.c/91
 102 2986 0             Right    126.6916   27326.1890  subfind_density()/subfind_density.c/92
 127 2986 0             Right    126.7763   27327.5514  subfind_density()/subfind_density.c/92
 127 2987 0              Todo     15.8470   27343.3985  subfind_density()/subfind_density.c/93
 127 2988 0             DM_Vx    126.7763   27470.1748  subfind_density()/subfind_density.c/95
 127 2989 0             DM_Vy    126.7763   27596.9511  subfind_density()/subfind_density.c/96
 125 2985 0              Left    126.7473   27200.7794  subfind_density()/subfind_density.c/91
 125 2986 0             Right    126.7473   27327.5267  subfind_density()/subfind_density.c/92
 125 2987 0              Todo     15.8434   27343.3701  subfind_density()/subfind_density.c/93
 125 2988 0             DM_Vx    126.7473   27470.1174  subfind_density()/subfind_density.c/95
 125 2989 0             DM_Vy    126.7473   27596.8647  subfind_density()/subfind_density.c/96
 125 2990 0             DM_Vz    126.7473   27723.6119  subfind_density()/subfind_density.c/97
   0 2973 0         FOF_PList    379.6103   18860.4614  fof_fof()/fof.c/249
  25 2991 0    	  DensDataGet    131.8250   27856.2403  subfind_density()/subfind_density.c/161
  25 2992 0  	  DensDataResult     98.8687   27955.1090  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 102 2987 0              Todo     15.8364   27342.0254  subfind_density()/subfind_density.c/93
 127 2990 0             DM_Vz    126.7763   27723.7274  subfind_density()/subfind_density.c/97
 127 2991 0    	  DensDataGet    118.9136   27842.6410  subfind_density()/subfind_density.c/161
 127 2992 0  	  DensDataResult     89.1852   27931.8262  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------- 125 2991 0    	  DensDataGet    110.8649   27834.4768  subfind_density()/subfind_density.c/161
 125 2992 0  	  DensDataResult     83.1487   27917.6255  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=125: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.3745 MB).
   0 2974 0         FOF_GList    632.6838   19493.1452  fof_fof()/fof.c/337

Task=25: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=44.891 MB).
 102 2988 0             DM_Vx    126.6916   27468.7170  subfind_density()/subfind_density.c/95


Task=127: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.1738 MB).
task 125: endrun called with an error level of 812


   0 2975 0             Group     12.4273   19505.5725  fof_fof()/fof.c/380
task 25: endrun called with an error level of 812


 102 2989 0             DM_Vy    126.6916   27595.4086  subfind_density()/subfind_density.c/96
task 127: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 125
   0 2976 0   DomainNodeIndex      0.3495   19505.9220  force_treeallocate()/forcetree.c/4800
   0 2977 0        Nodes_base   3737.6006   23243.5227  force_treeallocate()/forcetree.c/4803
   0 2978 0     Extnodes_base   3114.6672   26358.1899  force_treeallocate()/forcetree.c/4811
   0 2979 0          Nextnode    222.3050   26580.4949  force_treeallocate()/forcetree.c/4820
   0 2980 0            Father    221.9056   26802.4005  force_treeallocate()/forcetree.c/4827
   0 2981 0           Ngblist     63.2684   26865.6689  subfind_density()/subfind_density.c/78
   0 2982 0         Dist2list    126.5368   26992.2056  subfind_density()/subfind_density.c/79
   0 2983 0    DataIndexTable     21.8182   27014.0238  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 25
 102 2990 0             DM_Vz    126.6916   27722.1001  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 127
   0 2984 0      DataNodeList     58.1818   27072.2056  subfind_density()/subfind_density.c/89
   0 2985 0              Left    126.5368   27198.7424  subfind_density()/subfind_density.c/91
 102 2991 0    	  DensDataGet    117.7427   27839.8428  subfind_density()/subfind_density.c/161
   0 2986 0             Right    126.5368   27325.2791  subfind_density()/subfind_density.c/92
 102 2992 0  	  DensDataResult     88.3070   27928.1498  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   0 2987 0              Todo     15.8171   27341.0962  subfind_density()/subfind_density.c/93
   0 2988 0             DM_Vx    126.5368   27467.6330  subfind_density()/subfind_density.c/95
   0 2989 0             DM_Vy    126.5368   27594.1697  subfind_density()/subfind_density.c/96
   0 2990 0             DM_Vz    126.5368   27720.7065  subfind_density()/subfind_density.c/97
   0 2991 0    	  DensDataGet    115.4266   27836.1331  subfind_density()/subfind_density.c/161
   0 2992 0  	  DensDataResult     86.5700   27922.7031  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=102: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.8502 MB).
task 102: endrun called with an error level of 812



Task=0: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.2969 MB).
task 0: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 102
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 0
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 0: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 0
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  16 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  16 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  16 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  16 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  16 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  16 2973 0         FOF_PList    380.4655   18861.3177  fof_fof()/fof.c/249
 111 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 111 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 111 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 111 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 111 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 111 2973 0         FOF_PList    380.6126   18861.4647  fof_fof()/fof.c/249
 111 2974 0         FOF_GList    634.3543   19495.8190  fof_fof()/fof.c/337
 111 2975 0             Group     10.4777   19506.2967  fof_fof()/fof.c/380
 111 2976 0   DomainNodeIndex      0.3495   19506.6462  force_treeallocate()/forcetree.c/4800
 111 2977 0        Nodes_base   3737.6006   23244.2469  force_treeallocate()/forcetree.c/4803
 111 2978 0     Extnodes_base   3114.6672   26358.9141  force_treeallocate()/forcetree.c/4811
 111 2979 0          Nextnode    222.3050   26581.2191  force_treeallocate()/forcetree.c/4820
 111 2980 0            Father    221.9056   26803.1247  force_treeallocate()/forcetree.c/4827
 111 2981 0           Ngblist     63.4354   26866.5601  subfind_density()/subfind_density.c/78
 111 2982 0         Dist2list    126.8709   26993.4310  subfind_density()/subfind_density.c/79
 111 2983 0    DataIndexTable     21.8182   27015.2492  subfind_density()/subfind_density.c/87
 111 2984 0      DataNodeList     58.1818   27073.4309  subfind_density()/subfind_density.c/89
  16 2974 0         FOF_GList    634.1092   19495.4268  fof_fof()/fof.c/337
  16 2975 0             Group     11.8224   19507.2492  fof_fof()/fof.c/380
 111 2985 0              Left    126.8709   27200.3018  subfind_density()/subfind_density.c/91
 111 2986 0             Right    126.8709   27327.1727  subfind_density()/subfind_density.c/92
 111 2987 0              Todo     15.8589   27343.0315  subfind_density()/subfind_density.c/93
 111 2988 0             DM_Vx    126.8709   27469.9024  subfind_density()/subfind_density.c/95
 111 2989 0             DM_Vy    126.8709   27596.7732  subfind_density()/subfind_density.c/96
 111 2990 0             DM_Vz    126.8709   27723.6441  subfind_density()/subfind_density.c/97
  16 2976 0   DomainNodeIndex      0.3495   19507.5987  force_treeallocate()/forcetree.c/4800
  16 2977 0        Nodes_base   3737.6006   23245.1994  force_treeallocate()/forcetree.c/4803
  16 2978 0     Extnodes_base   3114.6672   26359.8666  force_treeallocate()/forcetree.c/4811
 111 2991 0    	  DensDataGet    113.4642   27837.1083  subfind_density()/subfind_density.c/161
 111 2992 0  	  DensDataResult     85.0981   27922.2064  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=111: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.7936 MB).
  16 2979 0          Nextnode    222.3050   26582.1716  force_treeallocate()/forcetree.c/4820
  16 2980 0            Father    221.9056   26804.0772  force_treeallocate()/forcetree.c/4827
task 111: endrun called with an error level of 812


  16 2981 0           Ngblist     63.4109   26867.4881  subfind_density()/subfind_density.c/78
  16 2982 0         Dist2list    126.8218   26994.3100  subfind_density()/subfind_density.c/79
  16 2983 0    DataIndexTable     21.8182   27016.1281  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 111
  16 2984 0      DataNodeList     58.1818   27074.3099  subfind_density()/subfind_density.c/89
  16 2985 0              Left    126.8218   27201.1318  subfind_density()/subfind_density.c/91
  16 2986 0             Right    126.8218   27327.9536  subfind_density()/subfind_density.c/92
  16 2987 0              Todo     15.8527   27343.8063  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2988 0             DM_Vx    126.8218   27470.6281  subfind_density()/subfind_density.c/95
  24 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  24 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  24 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  24 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2989 0             DM_Vy    126.8218   27597.4500  subfind_density()/subfind_density.c/96
  24 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  24 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 103 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 103 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 103 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 103 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 103 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 103 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 103 2973 0         FOF_PList    380.6215   18861.4737  fof_fof()/fof.c/249
 103 2974 0         FOF_GList    634.3692   19495.8429  fof_fof()/fof.c/337
 103 2975 0             Group     11.6118   19507.4547  fof_fof()/fof.c/380
 103 2976 0   DomainNodeIndex      0.3495   19507.8042  force_treeallocate()/forcetree.c/4800
 103 2977 0        Nodes_base   3737.6006   23245.4048  force_treeallocate()/forcetree.c/4803
 103 2978 0     Extnodes_base   3114.6672   26360.0720  force_treeallocate()/forcetree.c/4811
  16 2990 0             DM_Vz    126.8218   27724.2718  subfind_density()/subfind_density.c/97
  24 2973 0         FOF_PList    380.5258   18861.3780  fof_fof()/fof.c/249
  24 2974 0         FOF_GList    634.2096   19495.5876  fof_fof()/fof.c/337
 103 2979 0          Nextnode    222.3050   26582.3771  force_treeallocate()/forcetree.c/4820
 103 2980 0            Father    221.9056   26804.2826  force_treeallocate()/forcetree.c/4827
 103 2981 0           Ngblist     63.4369   26867.7196  subfind_density()/subfind_density.c/78
 103 2982 0         Dist2list    126.8738   26994.5934  subfind_density()/subfind_density.c/79
  16 2991 0    	  DensDataGet    114.2408   27838.5126  subfind_density()/subfind_density.c/161
  24 2975 0             Group     10.7101   19506.2977  fof_fof()/fof.c/380
  24 2976 0   DomainNodeIndex      0.3495   19506.6472  force_treeallocate()/forcetree.c/4800
 103 2983 0    DataIndexTable     21.8182   27016.4116  subfind_density()/subfind_density.c/87
 103 2984 0      DataNodeList     58.1818   27074.5934  subfind_density()/subfind_density.c/89
 103 2985 0              Left    126.8738   27201.4672  subfind_density()/subfind_density.c/91
  16 2992 0  	  DensDataResult     85.6806   27924.1933  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  24 2977 0        Nodes_base   3737.6006   23244.2478  force_treeallocate()/forcetree.c/4803
  24 2978 0     Extnodes_base   3114.6672   26358.9150  force_treeallocate()/forcetree.c/4811
 103 2986 0             Right    126.8738   27328.3411  subfind_density()/subfind_density.c/92
 103 2987 0              Todo     15.8592   27344.2003  subfind_density()/subfind_density.c/93
 103 2988 0             DM_Vx    126.8738   27471.0741  subfind_density()/subfind_density.c/95

Task=16: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.8067 MB).
  24 2979 0          Nextnode    222.3050   26581.2201  force_treeallocate()/forcetree.c/4820
  24 2980 0            Father    221.9056   26803.1257  force_treeallocate()/forcetree.c/4827
 103 2989 0             DM_Vy    126.8738   27597.9480  subfind_density()/subfind_density.c/96
 103 2990 0             DM_Vz    126.8738   27724.8218  subfind_density()/subfind_density.c/97
 103 2991 0    	  DensDataGet    125.9688   27850.7906  subfind_density()/subfind_density.c/161
task 16: endrun called with an error level of 812


  24 2981 0           Ngblist     63.4210   26866.5466  subfind_density()/subfind_density.c/78
 103 2992 0  	  DensDataResult     94.4766   27945.2671  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=103: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.7329 MB).
task 103: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 16
  24 2982 0         Dist2list    126.8419   26993.3886  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 103
 An HDF5 error was detected, we'd better stop.
  24 2983 0    DataIndexTable     21.8182   27015.2067  subfind_density()/subfind_density.c/87
HDF5-DIAG: Error detected in HDF5 (1.8.12)   24 2984 0      DataNodeList     58.1818   27073.3885  subfind_density()/subfind_density.c/89
MPI-process 16:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
  24 2985 0              Left    126.8419   27200.2305  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
task 16: endrun called with an error level of 124


  24 2986 0             Right    126.8419   27327.0724  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   9 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   9 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   9 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   9 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   9 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   9 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   9 2973 0         FOF_PList    380.6243   18861.4765  fof_fof()/fof.c/249
   9 2974 0         FOF_GList    634.3738   19495.8503  fof_fof()/fof.c/337
   9 2975 0             Group     12.5280   19508.3782  fof_fof()/fof.c/380
   9 2976 0   DomainNodeIndex      0.3495   19508.7278  force_treeallocate()/forcetree.c/4800
   9 2977 0        Nodes_base   3737.6006   23246.3284  force_treeallocate()/forcetree.c/4803
   9 2978 0     Extnodes_base   3114.6672   26360.9956  force_treeallocate()/forcetree.c/4811
   9 2979 0          Nextnode    222.3050   26583.3006  force_treeallocate()/forcetree.c/4820
   9 2980 0            Father    221.9056   26805.2062  force_treeallocate()/forcetree.c/4827
   9 2981 0           Ngblist     63.4374   26868.6436  subfind_density()/subfind_density.c/78
   9 2982 0         Dist2list    126.8748   26995.5184  subfind_density()/subfind_density.c/79
   9 2983 0    DataIndexTable     21.8182   27017.3366  subfind_density()/subfind_density.c/87
   9 2984 0      DataNodeList     58.1818   27075.5183  subfind_density()/subfind_density.c/89
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 16
  24 2987 0              Todo     15.8552   27342.9276  subfind_density()/subfind_density.c/93
 118 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 118 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 118 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 118 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 118 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 118 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   9 2985 0              Left    126.8748   27202.3931  subfind_density()/subfind_density.c/91
   9 2986 0             Right    126.8748   27329.2679  subfind_density()/subfind_density.c/92
   9 2987 0              Todo     15.8594   27345.1272  subfind_density()/subfind_density.c/93
   9 2988 0             DM_Vx    126.8748   27472.0020  subfind_density()/subfind_density.c/95
   9 2989 0             DM_Vy    126.8748   27598.8768  subfind_density()/subfind_density.c/96
   9 2990 0             DM_Vz    126.8748   27725.7515  subfind_density()/subfind_density.c/97
  24 2988 0             DM_Vx    126.8419   27469.7696  subfind_density()/subfind_density.c/95
 118 2973 0         FOF_PList    380.4524   18861.3046  fof_fof()/fof.c/249
 118 2974 0         FOF_GList    634.0873   19495.3918  fof_fof()/fof.c/337
 118 2975 0             Group     12.9783   19508.3701  fof_fof()/fof.c/380
   9 2991 0    	  DensDataGet    114.9043   27840.6558  subfind_density()/subfind_density.c/161
   9 2992 0  	  DensDataResult     86.1782   27926.8340  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  24 2989 0             DM_Vy    126.8419   27596.6115  subfind_density()/subfind_density.c/96
 118 2976 0   DomainNodeIndex      0.3495   19508.7196  force_treeallocate()/forcetree.c/4800
 118 2977 0        Nodes_base   3737.6006   23246.3203  force_treeallocate()/forcetree.c/4803
 118 2978 0     Extnodes_base   3114.6672   26360.9875  force_treeallocate()/forcetree.c/4811

Task=9: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.166 MB).
task 9: endrun called with an error level of 812


  24 2990 0             DM_Vz    126.8419   27723.4534  subfind_density()/subfind_density.c/97
 118 2979 0          Nextnode    222.3050   26583.2925  force_treeallocate()/forcetree.c/4820
 118 2980 0            Father    221.9056   26805.1981  force_treeallocate()/forcetree.c/4827
 118 2981 0           Ngblist     63.4087   26868.6068  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 9
  24 2991 0    	  DensDataGet    125.1658   27848.6192  subfind_density()/subfind_density.c/161
 118 2982 0         Dist2list    126.8175   26995.4243  subfind_density()/subfind_density.c/79
 118 2983 0    DataIndexTable     21.8182   27017.2425  subfind_density()/subfind_density.c/87
 118 2984 0      DataNodeList     58.1818   27075.4243  subfind_density()/subfind_density.c/89
  24 2992 0  	  DensDataResult     93.8743   27942.4935  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 118 2985 0              Left    126.8175   27202.2417  subfind_density()/subfind_density.c/91
 118 2986 0             Right    126.8175   27329.0592  subfind_density()/subfind_density.c/92
 118 2987 0              Todo     15.8522   27344.9114  subfind_density()/subfind_density.c/93

Task=24: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.5065 MB).
task 24: endrun called with an error level of 812


 118 2988 0             DM_Vx    126.8175   27471.7288  subfind_density()/subfind_density.c/95
 118 2989 0             DM_Vy    126.8175   27598.5463  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 24
 118 2990 0             DM_Vz    126.8175   27725.3638  subfind_density()/subfind_density.c/97
 113 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 113 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 113 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 113 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 113 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 113 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 113 2973 0         FOF_PList    380.5582   18861.4104  fof_fof()/fof.c/249
 113 2974 0         FOF_GList    634.2637   19495.6741  fof_fof()/fof.c/337
 113 2975 0             Group     12.5072   19508.1813  fof_fof()/fof.c/380
 113 2976 0   DomainNodeIndex      0.3495   19508.5308  force_treeallocate()/forcetree.c/4800
 113 2977 0        Nodes_base   3737.6006   23246.1314  force_treeallocate()/forcetree.c/4803
 113 2978 0     Extnodes_base   3114.6672   26360.7986  force_treeallocate()/forcetree.c/4811
 113 2979 0          Nextnode    222.3050   26583.1037  force_treeallocate()/forcetree.c/4820
 113 2980 0            Father    221.9056   26805.0093  force_treeallocate()/forcetree.c/4827
 An HDF5 error was detected, we'd better stop.
 118 2991 0    	  DensDataGet    109.6238   27834.9875  subfind_density()/subfind_density.c/161
 113 2981 0           Ngblist     63.4264   26868.4356  subfind_density()/subfind_density.c/78
 113 2982 0         Dist2list    126.8527   26995.2884  subfind_density()/subfind_density.c/79
 113 2983 0    DataIndexTable     21.8182   27017.1066  subfind_density()/subfind_density.c/87
 113 2984 0      DataNodeList     58.1818   27075.2883  subfind_density()/subfind_density.c/89
 113 2985 0              Left    126.8527   27202.1411  subfind_density()/subfind_density.c/91
HDF5-DIAG: Error detected in HDF5 (1.8.12)  118 2992 0  	  DensDataResult     82.2178   27917.2054  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 113 2986 0             Right    126.8527   27328.9938  subfind_density()/subfind_density.c/92
 113 2987 0              Todo     15.8566   27344.8504  subfind_density()/subfind_density.c/93
 113 2988 0             DM_Vx    126.8527   27471.7032  subfind_density()/subfind_density.c/95
MPI-process 24:

Task=118: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.7946 MB).
task 118: endrun called with an error level of 812


 113 2989 0             DM_Vy    126.8527   27598.5559  subfind_density()/subfind_density.c/96
 113 2990 0             DM_Vz    126.8527   27725.4086  subfind_density()/subfind_density.c/97
 113 2991 0    	  DensDataGet    126.6755   27852.0841  subfind_density()/subfind_density.c/161
task 24: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 118
 113 2992 0  	  DensDataResult     95.0066   27947.0907  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=113: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.9093 MB).
task 113: endrun called with an error level of 812


  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 24
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 113
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 112 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 112 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 112 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 112 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 112 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 112 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  15 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  15 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  15 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  15 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  15 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  15 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  15 2973 0         FOF_PList    380.4680   18861.3201  fof_fof()/fof.c/249
  15 2974 0         FOF_GList    634.1133   19495.4334  fof_fof()/fof.c/337
  15 2975 0             Group     12.0548   19507.4882  fof_fof()/fof.c/380
  15 2976 0   DomainNodeIndex      0.3495   19507.8377  force_treeallocate()/forcetree.c/4800
  15 2977 0        Nodes_base   3737.6006   23245.4384  force_treeallocate()/forcetree.c/4803
  15 2978 0     Extnodes_base   3114.6672   26360.1056  force_treeallocate()/forcetree.c/4811
  15 2979 0          Nextnode    222.3050   26582.4106  force_treeallocate()/forcetree.c/4820
  15 2980 0            Father    221.9056   26804.3162  force_treeallocate()/forcetree.c/4827
 112 2973 0         FOF_PList    380.4277   18861.2799  fof_fof()/fof.c/249
 112 2974 0         FOF_GList    634.0462   19495.3262  fof_fof()/fof.c/337
  15 2981 0           Ngblist     63.4113   26867.7275  subfind_density()/subfind_density.c/78
  15 2982 0         Dist2list    126.8227   26994.5502  subfind_density()/subfind_density.c/79
  15 2983 0    DataIndexTable     21.8182   27016.3684  subfind_density()/subfind_density.c/87
  15 2984 0      DataNodeList     58.1818   27074.5502  subfind_density()/subfind_density.c/89
  15 2985 0              Left    126.8227   27201.3728  subfind_density()/subfind_density.c/91
 112 2975 0             Group     10.9550   19506.2811  fof_fof()/fof.c/380
 112 2976 0   DomainNodeIndex      0.3495   19506.6306  force_treeallocate()/forcetree.c/4800
 112 2977 0        Nodes_base   3737.6006   23244.2313  force_treeallocate()/forcetree.c/4803
  15 2986 0             Right    126.8227   27328.1955  subfind_density()/subfind_density.c/92
  15 2987 0              Todo     15.8528   27344.0483  subfind_density()/subfind_density.c/93
  15 2988 0             DM_Vx    126.8227   27470.8710  subfind_density()/subfind_density.c/95
 112 2978 0     Extnodes_base   3114.6672   26358.8985  force_treeallocate()/forcetree.c/4811
 112 2979 0          Nextnode    222.3050   26581.2035  force_treeallocate()/forcetree.c/4820
 112 2980 0            Father    221.9056   26803.1091  force_treeallocate()/forcetree.c/4827
  15 2989 0             DM_Vy    126.8227   27597.6936  subfind_density()/subfind_density.c/96
  15 2990 0             DM_Vz    126.8227   27724.5163  subfind_density()/subfind_density.c/97
  15 2991 0    	  DensDataGet    137.2526   27861.7689  subfind_density()/subfind_density.c/161
 112 2981 0           Ngblist     63.4046   26866.5137  subfind_density()/subfind_density.c/78
 112 2982 0         Dist2list    126.8092   26993.3230  subfind_density()/subfind_density.c/79
 112 2983 0    DataIndexTable     21.8182   27015.1412  subfind_density()/subfind_density.c/87
  15 2992 0  	  DensDataResult    102.9395   27964.7084  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=15: Not enough memory in mymalloc_fullinfo() to allocate 87.2725 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=35.2916 MB).
task 15: endrun called with an error level of 812


 112 2984 0      DataNodeList     58.1818   27073.3230  subfind_density()/subfind_density.c/89
 112 2985 0              Left    126.8092   27200.1322  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 15
 112 2986 0             Right    126.8092   27326.9415  subfind_density()/subfind_density.c/92
 112 2987 0              Todo     15.8512   27342.7926  subfind_density()/subfind_density.c/93
 112 2988 0             DM_Vx    126.8092   27469.6019  subfind_density()/subfind_density.c/95
 112 2989 0             DM_Vy    126.8092   27596.4111  subfind_density()/subfind_density.c/96
 112 2990 0             DM_Vz    126.8092   27723.2204  subfind_density()/subfind_density.c/97
 112 2991 0    	  DensDataGet    115.9861   27839.2065  subfind_density()/subfind_density.c/161
 112 2992 0  	  DensDataResult     86.9896   27926.1961  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=112: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.8039 MB).
task 112: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 112
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 112:
task 112: endrun called with an error level of 124


  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 112
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 124 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 124 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 124 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 124 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 124 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 124 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 124 2973 0         FOF_PList    379.5862   18860.4384  fof_fof()/fof.c/249
   3 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   3 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   3 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   3 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   3 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   3 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
   3 2973 0         FOF_PList    380.5926   18861.4447  fof_fof()/fof.c/249
   3 2974 0         FOF_GList    634.3209   19495.7657  fof_fof()/fof.c/337
   3 2975 0             Group     12.4813   19508.2469  fof_fof()/fof.c/380
   3 2976 0   DomainNodeIndex      0.3495   19508.5965  force_treeallocate()/forcetree.c/4800
   3 2977 0        Nodes_base   3737.6006   23246.1971  force_treeallocate()/forcetree.c/4803
   3 2978 0     Extnodes_base   3114.6672   26360.8643  force_treeallocate()/forcetree.c/4811
   3 2979 0          Nextnode    222.3050   26583.1694  force_treeallocate()/forcetree.c/4820
   3 2980 0            Father    221.9056   26805.0749  force_treeallocate()/forcetree.c/4827
   3 2981 0           Ngblist     63.4321   26868.5070  subfind_density()/subfind_density.c/78
   3 2982 0         Dist2list    126.8642   26995.3712  subfind_density()/subfind_density.c/79
   3 2983 0    DataIndexTable     21.8182   27017.1894  subfind_density()/subfind_density.c/87
   3 2984 0      DataNodeList     58.1818   27075.3712  subfind_density()/subfind_density.c/89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 124 2974 0         FOF_GList    632.6437   19493.0822  fof_fof()/fof.c/337
 124 2975 0             Group     10.3397   19503.4218  fof_fof()/fof.c/380
 124 2976 0   DomainNodeIndex      0.3495   19503.7713  force_treeallocate()/forcetree.c/4800
 124 2977 0        Nodes_base   3737.6006   23241.3720  force_treeallocate()/forcetree.c/4803
   3 2985 0              Left    126.8642   27202.2354  subfind_density()/subfind_density.c/91
   3 2986 0             Right    126.8642   27329.0996  subfind_density()/subfind_density.c/92
   3 2987 0              Todo     15.8580   27344.9576  subfind_density()/subfind_density.c/93
   3 2988 0             DM_Vx    126.8642   27471.8218  subfind_density()/subfind_density.c/95
   3 2989 0             DM_Vy    126.8642   27598.6860  subfind_density()/subfind_density.c/96
   3 2990 0             DM_Vz    126.8642   27725.5502  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  26 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  26 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  26 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  26 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  26 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  26 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  26 2973 0         FOF_PList    380.5593   18861.4115  fof_fof()/fof.c/249
  26 2974 0         FOF_GList    634.2655   19495.6769  fof_fof()/fof.c/337
  26 2975 0             Group     11.8919   19507.5688  fof_fof()/fof.c/380
  26 2976 0   DomainNodeIndex      0.3495   19507.9184  force_treeallocate()/forcetree.c/4800
  26 2977 0        Nodes_base   3737.6006   23245.5190  force_treeallocate()/forcetree.c/4803
  26 2978 0     Extnodes_base   3114.6672   26360.1862  force_treeallocate()/forcetree.c/4811
 124 2978 0     Extnodes_base   3114.6672   26356.0392  force_treeallocate()/forcetree.c/4811
 124 2979 0          Nextnode    222.3050   26578.3442  force_treeallocate()/forcetree.c/4820
 124 2980 0            Father    221.9056   26800.2498  force_treeallocate()/forcetree.c/4827
   3 2991 0    	  DensDataGet    113.6314   27839.1816  subfind_density()/subfind_density.c/161
   3 2992 0  	  DensDataResult     85.2236   27924.4051  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=3: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.5949 MB).
 101 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 101 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 101 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 101 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 101 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 101 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 101 2973 0         FOF_PList    380.5676   18861.4198  fof_fof()/fof.c/249
 101 2974 0         FOF_GList    634.2794   19495.6992  fof_fof()/fof.c/337
 101 2975 0             Group     11.5744   19507.2736  fof_fof()/fof.c/380
 101 2976 0   DomainNodeIndex      0.3495   19507.6231  force_treeallocate()/forcetree.c/4800
 101 2977 0        Nodes_base   3737.6006   23245.2237  force_treeallocate()/forcetree.c/4803
 101 2978 0     Extnodes_base   3114.6672   26359.8909  force_treeallocate()/forcetree.c/4811
 101 2979 0          Nextnode    222.3050   26582.1960  force_treeallocate()/forcetree.c/4820
 101 2980 0            Father    221.9056   26804.1016  force_treeallocate()/forcetree.c/4827
 101 2981 0           Ngblist     63.4279   26867.5295  subfind_density()/subfind_density.c/78
  26 2979 0          Nextnode    222.3050   26582.4913  force_treeallocate()/forcetree.c/4820
  26 2980 0            Father    221.9056   26804.3968  force_treeallocate()/forcetree.c/4827
 124 2981 0           Ngblist     63.2644   26863.5142  subfind_density()/subfind_density.c/78
 124 2982 0         Dist2list    126.5287   26990.0429  subfind_density()/subfind_density.c/79
 124 2983 0    DataIndexTable     21.8182   27011.8611  subfind_density()/subfind_density.c/87
task 3: endrun called with an error level of 812


 101 2982 0         Dist2list    126.8559   26994.3854  subfind_density()/subfind_density.c/79
 101 2983 0    DataIndexTable     21.8182   27016.2035  subfind_density()/subfind_density.c/87
 101 2984 0      DataNodeList     58.1818   27074.3853  subfind_density()/subfind_density.c/89
 101 2985 0              Left    126.8559   27201.2412  subfind_density()/subfind_density.c/91
 101 2986 0             Right    126.8559   27328.0971  subfind_density()/subfind_density.c/92
  26 2981 0           Ngblist     63.4266   26867.8234  subfind_density()/subfind_density.c/78
  26 2982 0         Dist2list    126.8531   26994.6765  subfind_density()/subfind_density.c/79
  26 2983 0    DataIndexTable     21.8182   27016.4947  subfind_density()/subfind_density.c/87
 124 2984 0      DataNodeList     58.1818   27070.0429  subfind_density()/subfind_density.c/89
 124 2985 0              Left    126.5287   27196.5716  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 3
 101 2987 0              Todo     15.8570   27343.9541  subfind_density()/subfind_density.c/93
 101 2988 0             DM_Vx    126.8559   27470.8099  subfind_density()/subfind_density.c/95
 101 2989 0             DM_Vy    126.8559   27597.6658  subfind_density()/subfind_density.c/96
  26 2984 0      DataNodeList     58.1818   27074.6765  subfind_density()/subfind_density.c/89
  26 2985 0              Left    126.8531   27201.5296  subfind_density()/subfind_density.c/91
  26 2986 0             Right    126.8531   27328.3827  subfind_density()/subfind_density.c/92
  26 2987 0              Todo     15.8566   27344.2393  subfind_density()/subfind_density.c/93
 124 2986 0             Right    126.5287   27323.1004  subfind_density()/subfind_density.c/92
 101 2990 0             DM_Vz    126.8559   27724.5217  subfind_density()/subfind_density.c/97
 101 2991 0    	  DensDataGet    121.1411   27845.6628  subfind_density()/subfind_density.c/161
 101 2992 0  	  DensDataResult     90.8558   27936.5186  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  26 2988 0             DM_Vx    126.8531   27471.0924  subfind_density()/subfind_density.c/95
  26 2989 0             DM_Vy    126.8531   27597.9455  subfind_density()/subfind_density.c/96
 124 2987 0              Todo     15.8161   27338.9165  subfind_density()/subfind_density.c/93
 124 2988 0             DM_Vx    126.5287   27465.4452  subfind_density()/subfind_density.c/95
 124 2989 0             DM_Vy    126.5287   27591.9740  subfind_density()/subfind_density.c/96
 124 2990 0             DM_Vz    126.5287   27718.5027  subfind_density()/subfind_density.c/97

Task=101: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=63.4814 MB).
task 101: endrun called with an error level of 812


  26 2990 0             DM_Vz    126.8531   27724.7986  subfind_density()/subfind_density.c/97
  26 2991 0    	  DensDataGet    136.6832   27861.4818  subfind_density()/subfind_density.c/161
 124 2991 0    	  DensDataGet    118.9835   27837.4862  subfind_density()/subfind_density.c/161
 124 2992 0  	  DensDataResult     89.2376   27926.7238  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 101
  26 2992 0  	  DensDataResult    102.5124   27963.9942  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=124: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.2762 MB).
task 124: endrun called with an error level of 812



Task=26: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=36.0058 MB).
task 26: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 124
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 26
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 116 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 116 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 116 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 116 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 116 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 116 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 116 2973 0         FOF_PList    380.6010   18861.4532  fof_fof()/fof.c/249
 116 2974 0         FOF_GList    634.3350   19495.7881  fof_fof()/fof.c/337
 116 2975 0             Group     10.3397   19506.1278  fof_fof()/fof.c/380
 116 2976 0   DomainNodeIndex      0.3495   19506.4773  force_treeallocate()/forcetree.c/4800
 116 2977 0        Nodes_base   3737.6006   23244.0780  force_treeallocate()/forcetree.c/4803
 116 2978 0     Extnodes_base   3114.6672   26358.7452  force_treeallocate()/forcetree.c/4811
  11 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  11 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  11 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  11 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  11 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  11 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  11 2973 0         FOF_PList    380.2757   18861.1278  fof_fof()/fof.c/249
  11 2974 0         FOF_GList    633.7928   19494.9206  fof_fof()/fof.c/337
  11 2975 0             Group     12.4958   19507.4164  fof_fof()/fof.c/380
  11 2976 0   DomainNodeIndex      0.3495   19507.7659  force_treeallocate()/forcetree.c/4800
  11 2977 0        Nodes_base   3737.6006   23245.3665  force_treeallocate()/forcetree.c/4803
  11 2978 0     Extnodes_base   3114.6672   26360.0338  force_treeallocate()/forcetree.c/4811
  11 2979 0          Nextnode    222.3050   26582.3388  force_treeallocate()/forcetree.c/4820
  11 2980 0            Father    221.9056   26804.2444  force_treeallocate()/forcetree.c/4827
  11 2981 0           Ngblist     63.3793   26867.6237  subfind_density()/subfind_density.c/78
  11 2982 0         Dist2list    126.7586   26994.3822  subfind_density()/subfind_density.c/79
  11 2983 0    DataIndexTable     21.8182   27016.2004  subfind_density()/subfind_density.c/87
  11 2984 0      DataNodeList     58.1818   27074.3822  subfind_density()/subfind_density.c/89
 116 2979 0          Nextnode    222.3050   26581.0502  force_treeallocate()/forcetree.c/4820
 116 2980 0            Father    221.9056   26802.9558  force_treeallocate()/forcetree.c/4827
 116 2981 0           Ngblist     63.4335   26866.3893  subfind_density()/subfind_density.c/78
  11 2985 0              Left    126.7586   27201.1407  subfind_density()/subfind_density.c/91
  11 2986 0             Right    126.7586   27327.8993  subfind_density()/subfind_density.c/92
  11 2987 0              Todo     15.8448   27343.7441  subfind_density()/subfind_density.c/93
  11 2988 0             DM_Vx    126.7586   27470.5027  subfind_density()/subfind_density.c/95
  11 2989 0             DM_Vy    126.7586   27597.2612  subfind_density()/subfind_density.c/96
  11 2990 0             DM_Vz    126.7586   27724.0198  subfind_density()/subfind_density.c/97
 116 2982 0         Dist2list    126.8670   26993.2563  subfind_density()/subfind_density.c/79
 116 2983 0    DataIndexTable     21.8182   27015.0745  subfind_density()/subfind_density.c/87
 116 2984 0      DataNodeList     58.1818   27073.2563  subfind_density()/subfind_density.c/89
  11 2991 0    	  DensDataGet    127.3893   27851.4091  subfind_density()/subfind_density.c/161
  11 2992 0  	  DensDataResult     95.5420   27946.9511  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=11: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=53.0489 MB).
 116 2985 0              Left    126.8670   27200.1233  subfind_density()/subfind_density.c/91
 116 2986 0             Right    126.8670   27326.9903  subfind_density()/subfind_density.c/92
task 11: endrun called with an error level of 812


 116 2987 0              Todo     15.8584   27342.8486  subfind_density()/subfind_density.c/93
 116 2988 0             DM_Vx    126.8670   27469.7156  subfind_density()/subfind_density.c/95
 116 2989 0             DM_Vy    126.8670   27596.5826  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 11
 116 2990 0             DM_Vz    126.8670   27723.4496  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 116 2991 0    	  DensDataGet    116.1138   27839.5635  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 108 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 108 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 108 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 108 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 108 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 108 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 108 2973 0         FOF_PList    379.7836   18860.6358  fof_fof()/fof.c/249
 108 2974 0         FOF_GList    632.9726   19493.6084  fof_fof()/fof.c/337
 108 2975 0             Group     10.8865   19504.4949  fof_fof()/fof.c/380
 108 2976 0   DomainNodeIndex      0.3495   19504.8444  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  28 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  28 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  28 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  28 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  28 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  28 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  28 2973 0         FOF_PList    380.6353   18861.4874  fof_fof()/fof.c/249
  28 2974 0         FOF_GList    634.3921   19495.8796  fof_fof()/fof.c/337
 116 2992 0  	  DensDataResult     87.0854   27926.6488  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  19 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  19 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  19 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  19 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  19 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  19 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  19 2973 0         FOF_PList    380.1989   18861.0510  fof_fof()/fof.c/249
  19 2974 0         FOF_GList    633.6648   19494.7158  fof_fof()/fof.c/337
  19 2975 0             Group     10.3397   19505.0555  fof_fof()/fof.c/380
  19 2976 0   DomainNodeIndex      0.3495   19505.4050  force_treeallocate()/forcetree.c/4800
  19 2977 0        Nodes_base   3737.6006   23243.0056  force_treeallocate()/forcetree.c/4803
  19 2978 0     Extnodes_base   3114.6672   26357.6729  force_treeallocate()/forcetree.c/4811
  19 2979 0          Nextnode    222.3050   26579.9779  force_treeallocate()/forcetree.c/4820
  19 2980 0            Father    221.9056   26801.8835  force_treeallocate()/forcetree.c/4827
  19 2981 0           Ngblist     63.3665   26865.2500  subfind_density()/subfind_density.c/78
  19 2982 0         Dist2list    126.7330   26991.9829  subfind_density()/subfind_density.c/79
 108 2977 0        Nodes_base   3737.6006   23242.4450  force_treeallocate()/forcetree.c/4803
 108 2978 0     Extnodes_base   3114.6672   26357.1122  force_treeallocate()/forcetree.c/4811
 108 2979 0          Nextnode    222.3050   26579.4173  force_treeallocate()/forcetree.c/4820
  99 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  99 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  99 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  99 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  99 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  99 2973 0         FOF_PList    380.6060   18861.4582  fof_fof()/fof.c/249
  99 2974 0         FOF_GList    634.3434   19495.8016  fof_fof()/fof.c/337
  99 2975 0             Group     11.6377   19507.4393  fof_fof()/fof.c/380
  99 2976 0   DomainNodeIndex      0.3495   19507.7888  force_treeallocate()/forcetree.c/4800
  99 2977 0        Nodes_base   3737.6006   23245.3895  force_treeallocate()/forcetree.c/4803
  99 2978 0     Extnodes_base   3114.6672   26360.0567  force_treeallocate()/forcetree.c/4811
  99 2979 0          Nextnode    222.3050   26582.3617  force_treeallocate()/forcetree.c/4820
  99 2980 0            Father    221.9056   26804.2673  force_treeallocate()/forcetree.c/4827
  99 2981 0           Ngblist     63.4343   26867.7017  subfind_density()/subfind_density.c/78
  28 2975 0             Group     13.2750   19509.1546  fof_fof()/fof.c/380
  28 2976 0   DomainNodeIndex      0.3495   19509.5041  force_treeallocate()/forcetree.c/4800

Task=116: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.3512 MB).
task 116: endrun called with an error level of 812


  19 2983 0    DataIndexTable     21.8182   27013.8011  subfind_density()/subfind_density.c/87
  19 2984 0      DataNodeList     58.1818   27071.9829  subfind_density()/subfind_density.c/89
  19 2985 0              Left    126.7330   27198.7158  subfind_density()/subfind_density.c/91
  19 2986 0             Right    126.7330   27325.4488  subfind_density()/subfind_density.c/92
 108 2980 0            Father    221.9056   26801.3229  force_treeallocate()/forcetree.c/4827
 108 2981 0           Ngblist     63.2973   26864.6201  subfind_density()/subfind_density.c/78
 108 2982 0         Dist2list    126.5945   26991.2147  subfind_density()/subfind_density.c/79
  99 2982 0         Dist2list    126.8687   26994.5704  subfind_density()/subfind_density.c/79
  99 2983 0    DataIndexTable     21.8182   27016.3885  subfind_density()/subfind_density.c/87
  99 2984 0      DataNodeList     58.1818   27074.5703  subfind_density()/subfind_density.c/89
  99 2985 0              Left    126.8687   27201.4390  subfind_density()/subfind_density.c/91
  99 2986 0             Right    126.8687   27328.3077  subfind_density()/subfind_density.c/92
  28 2977 0        Nodes_base   3737.6006   23247.1048  force_treeallocate()/forcetree.c/4803
  28 2978 0     Extnodes_base   3114.6672   26361.7720  force_treeallocate()/forcetree.c/4811
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 116
  19 2987 0              Todo     15.8416   27341.2904  subfind_density()/subfind_density.c/93
  19 2988 0             DM_Vx    126.7330   27468.0234  subfind_density()/subfind_density.c/95
  19 2989 0             DM_Vy    126.7330   27594.7563  subfind_density()/subfind_density.c/96
  19 2990 0             DM_Vz    126.7330   27721.4893  subfind_density()/subfind_density.c/97
 108 2983 0    DataIndexTable     21.8182   27013.0328  subfind_density()/subfind_density.c/87
 108 2984 0      DataNodeList     58.1818   27071.2146  subfind_density()/subfind_density.c/89
  99 2987 0              Todo     15.8586   27344.1663  subfind_density()/subfind_density.c/93
  99 2988 0             DM_Vx    126.8687   27471.0350  subfind_density()/subfind_density.c/95
  99 2989 0             DM_Vy    126.8687   27597.9036  subfind_density()/subfind_density.c/96
  28 2979 0          Nextnode    222.3050   26584.0770  force_treeallocate()/forcetree.c/4820
  28 2980 0            Father    221.9056   26805.9826  force_treeallocate()/forcetree.c/4827
  28 2981 0           Ngblist     63.4392   26869.4218  subfind_density()/subfind_density.c/78
  19 2991 0    	  DensDataGet    113.6979   27835.1872  subfind_density()/subfind_density.c/161
  19 2992 0  	  DensDataResult     85.2734   27920.4606  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 108 2985 0              Left    126.5945   27197.8092  subfind_density()/subfind_density.c/91
 108 2986 0             Right    126.5945   27324.4037  subfind_density()/subfind_density.c/92
 108 2987 0              Todo     15.8243   27340.2280  subfind_density()/subfind_density.c/93
  99 2990 0             DM_Vz    126.8687   27724.7723  subfind_density()/subfind_density.c/97
  99 2991 0    	  DensDataGet    117.9251   27842.6974  subfind_density()/subfind_density.c/161
  99 2992 0  	  DensDataResult     88.4438   27931.1413  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  28 2982 0         Dist2list    126.8784   26996.3002  subfind_density()/subfind_density.c/79
  28 2983 0    DataIndexTable     21.8182   27018.1184  subfind_density()/subfind_density.c/87
  28 2984 0      DataNodeList     58.1818   27076.3002  subfind_density()/subfind_density.c/89

Task=19: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.5394 MB).
task 19: endrun called with an error level of 812


 108 2988 0             DM_Vx    126.5945   27466.8225  subfind_density()/subfind_density.c/95

Task=99: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.8587 MB).
task 99: endrun called with an error level of 812


  28 2985 0              Left    126.8784   27203.1786  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 19
 108 2989 0             DM_Vy    126.5945   27593.4171  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 99
  28 2986 0             Right    126.8784   27330.0571  subfind_density()/subfind_density.c/92
 108 2990 0             DM_Vz    126.5945   27720.0116  subfind_density()/subfind_density.c/97
  28 2987 0              Todo     15.8598   27345.9169  subfind_density()/subfind_density.c/93
 108 2991 0    	  DensDataGet    116.8987   27836.9103  subfind_density()/subfind_density.c/161
  28 2988 0             DM_Vx    126.8784   27472.7953  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 108 2992 0  	  DensDataResult     87.6741   27924.5844  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  28 2989 0             DM_Vy    126.8784   27599.6737  subfind_density()/subfind_density.c/96
  18 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  18 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  18 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  18 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  18 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  18 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  18 2973 0         FOF_PList    380.5050   18861.3572  fof_fof()/fof.c/249

Task=108: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.4156 MB).
task 108: endrun called with an error level of 812


  28 2990 0             DM_Vz    126.8784   27726.5521  subfind_density()/subfind_density.c/97
  18 2974 0         FOF_GList    634.1751   19495.5323  fof_fof()/fof.c/337
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 108
  28 2991 0    	  DensDataGet    112.4359   27838.9881  subfind_density()/subfind_density.c/161
  18 2975 0             Group     11.0349   19506.5671  fof_fof()/fof.c/380
  18 2976 0   DomainNodeIndex      0.3495   19506.9167  force_treeallocate()/forcetree.c/4800
  18 2977 0        Nodes_base   3737.6006   23244.5173  force_treeallocate()/forcetree.c/4803
  28 2992 0  	  DensDataResult     84.3269   27923.3150  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  18 2978 0     Extnodes_base   3114.6672   26359.1845  force_treeallocate()/forcetree.c/4811
  18 2979 0          Nextnode    222.3050   26581.4895  force_treeallocate()/forcetree.c/4820
  18 2980 0            Father    221.9056   26803.3951  force_treeallocate()/forcetree.c/4827

  18 2981 0           Ngblist     63.4175   26866.8126  subfind_density()/subfind_density.c/78
  18 2982 0         Dist2list    126.8350   26993.6477  subfind_density()/subfind_density.c/79

Task=28: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.685 MB).
task 28: endrun called with an error level of 812


  18 2983 0    DataIndexTable     21.8182   27015.4658  subfind_density()/subfind_density.c/87
  18 2984 0      DataNodeList     58.1818   27073.6476  subfind_density()/subfind_density.c/89
  18 2985 0              Left    126.8350   27200.4826  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 28
  18 2986 0             Right    126.8350   27327.3177  subfind_density()/subfind_density.c/92
  18 2987 0              Todo     15.8544   27343.1720  subfind_density()/subfind_density.c/93
  18 2988 0             DM_Vx    126.8350   27470.0070  subfind_density()/subfind_density.c/95
  18 2989 0             DM_Vy    126.8350   27596.8421  subfind_density()/subfind_density.c/96
  18 2990 0             DM_Vz    126.8350   27723.6771  subfind_density()/subfind_density.c/97
  18 2991 0    	  DensDataGet    116.0338   27839.7108  subfind_density()/subfind_density.c/161
  18 2992 0  	  DensDataResult     87.0253   27926.7361  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=18: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.2639 MB).
task 18: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 18
  10 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  10 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  10 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  10 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  10 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  10 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 117 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 117 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 117 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 117 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 117 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 117 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 117 2973 0         FOF_PList    380.5205   18861.3727  fof_fof()/fof.c/249
 117 2974 0         FOF_GList    634.2008   19495.5735  fof_fof()/fof.c/337
 117 2975 0             Group     10.4673   19506.0408  fof_fof()/fof.c/380
 117 2976 0   DomainNodeIndex      0.3495   19506.3903  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  10 2973 0         FOF_PList    380.6220   18861.4742  fof_fof()/fof.c/249
  10 2974 0         FOF_GList    634.3700   19495.8441  fof_fof()/fof.c/337
  10 2975 0             Group     11.5350   19507.3791  fof_fof()/fof.c/380
 117 2977 0        Nodes_base   3737.6006   23243.9909  force_treeallocate()/forcetree.c/4803
 117 2978 0     Extnodes_base   3114.6672   26358.6581  force_treeallocate()/forcetree.c/4811
 117 2979 0          Nextnode    222.3050   26580.9632  force_treeallocate()/forcetree.c/4820
 117 2980 0            Father    221.9056   26802.8688  force_treeallocate()/forcetree.c/4827
  98 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  98 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  98 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  98 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  98 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  98 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  10 2976 0   DomainNodeIndex      0.3495   19507.7286  force_treeallocate()/forcetree.c/4800
  10 2977 0        Nodes_base   3737.6006   23245.3293  force_treeallocate()/forcetree.c/4803
  10 2978 0     Extnodes_base   3114.6672   26359.9965  force_treeallocate()/forcetree.c/4811
  10 2979 0          Nextnode    222.3050   26582.3015  force_treeallocate()/forcetree.c/4820
  10 2980 0            Father    221.9056   26804.2071  force_treeallocate()/forcetree.c/4827
 117 2981 0           Ngblist     63.4201   26866.2888  subfind_density()/subfind_density.c/78
 117 2982 0         Dist2list    126.8402   26993.1290  subfind_density()/subfind_density.c/79
 117 2983 0    DataIndexTable     21.8182   27014.9472  subfind_density()/subfind_density.c/87
 117 2984 0      DataNodeList     58.1818   27073.1290  subfind_density()/subfind_density.c/89
  98 2973 0         FOF_PList    380.6218   18861.4739  fof_fof()/fof.c/249
  98 2974 0         FOF_GList    634.3696   19495.8436  fof_fof()/fof.c/337
  98 2975 0             Group     11.0722   19506.9158  fof_fof()/fof.c/380
  10 2981 0           Ngblist     63.4370   26867.6441  subfind_density()/subfind_density.c/78
  10 2982 0         Dist2list    126.8740   26994.5181  subfind_density()/subfind_density.c/79
  10 2983 0    DataIndexTable     21.8182   27016.3363  subfind_density()/subfind_density.c/87
 117 2985 0              Left    126.8402   27199.9691  subfind_density()/subfind_density.c/91
 117 2986 0             Right    126.8402   27326.8093  subfind_density()/subfind_density.c/92
 117 2987 0              Todo     15.8550   27342.6643  subfind_density()/subfind_density.c/93
  98 2976 0   DomainNodeIndex      0.3495   19507.2653  force_treeallocate()/forcetree.c/4800
  98 2977 0        Nodes_base   3737.6006   23244.8659  force_treeallocate()/forcetree.c/4803
  10 2984 0      DataNodeList     58.1818   27074.5180  subfind_density()/subfind_density.c/89
  10 2985 0              Left    126.8740   27201.3920  subfind_density()/subfind_density.c/91
  10 2986 0             Right    126.8740   27328.2660  subfind_density()/subfind_density.c/92
  29 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  29 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  29 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  29 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  29 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  29 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  29 2973 0         FOF_PList    380.6200   18861.4721  fof_fof()/fof.c/249
  29 2974 0         FOF_GList    634.3666   19495.8387  fof_fof()/fof.c/337
  29 2975 0             Group     12.5446   19508.3833  fof_fof()/fof.c/380
  29 2976 0   DomainNodeIndex      0.3495   19508.7328  force_treeallocate()/forcetree.c/4800
  29 2977 0        Nodes_base   3737.6006   23246.3335  force_treeallocate()/forcetree.c/4803
 117 2988 0             DM_Vx    126.8402   27469.5045  subfind_density()/subfind_density.c/95
 117 2989 0             DM_Vy    126.8402   27596.3447  subfind_density()/subfind_density.c/96
 117 2990 0             DM_Vz    126.8402   27723.1848  subfind_density()/subfind_density.c/97
  98 2978 0     Extnodes_base   3114.6672   26359.5331  force_treeallocate()/forcetree.c/4811
  98 2979 0          Nextnode    222.3050   26581.8382  force_treeallocate()/forcetree.c/4820
  98 2980 0            Father    221.9056   26803.7438  force_treeallocate()/forcetree.c/4827
  10 2987 0              Todo     15.8593   27344.1253  subfind_density()/subfind_density.c/93
  10 2988 0             DM_Vx    126.8740   27470.9993  subfind_density()/subfind_density.c/95
  29 2978 0     Extnodes_base   3114.6672   26361.0007  force_treeallocate()/forcetree.c/4811
  29 2979 0          Nextnode    222.3050   26583.3057  force_treeallocate()/forcetree.c/4820
  29 2980 0            Father    221.9056   26805.2113  force_treeallocate()/forcetree.c/4827
  29 2981 0           Ngblist     63.4367   26868.6480  subfind_density()/subfind_density.c/78
 117 2991 0    	  DensDataGet    116.6914   27839.8762  subfind_density()/subfind_density.c/161
 117 2992 0  	  DensDataResult     87.5186   27927.3948  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  98 2981 0           Ngblist     63.4370   26867.1807  subfind_density()/subfind_density.c/78
  98 2982 0         Dist2list    126.8739   26994.0547  subfind_density()/subfind_density.c/79
  98 2983 0    DataIndexTable     21.8182   27015.8728  subfind_density()/subfind_density.c/87
  10 2989 0             DM_Vy    126.8740   27597.8733  subfind_density()/subfind_density.c/96
  29 2982 0         Dist2list    126.8733   26995.5213  subfind_density()/subfind_density.c/79
  29 2983 0    DataIndexTable     21.8182   27017.3395  subfind_density()/subfind_density.c/87
  29 2984 0      DataNodeList     58.1818   27075.5212  subfind_density()/subfind_density.c/89

Task=117: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.6052 MB).
task 117: endrun called with an error level of 812


  98 2984 0      DataNodeList     58.1818   27074.0546  subfind_density()/subfind_density.c/89
  98 2985 0              Left    126.8739   27200.9286  subfind_density()/subfind_density.c/91
  98 2986 0             Right    126.8739   27327.8025  subfind_density()/subfind_density.c/92
  10 2990 0             DM_Vz    126.8740   27724.7473  subfind_density()/subfind_density.c/97
  29 2985 0              Left    126.8733   27202.3946  subfind_density()/subfind_density.c/91
  29 2986 0             Right    126.8733   27329.2679  subfind_density()/subfind_density.c/92
  29 2987 0              Todo     15.8592   27345.1271  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 117
  98 2987 0              Todo     15.8592   27343.6617  subfind_density()/subfind_density.c/93
  10 2991 0    	  DensDataGet    124.6198   27849.3670  subfind_density()/subfind_density.c/161
  29 2988 0             DM_Vx    126.8733   27472.0004  subfind_density()/subfind_density.c/95
  29 2989 0             DM_Vy    126.8733   27598.8737  subfind_density()/subfind_density.c/96
  98 2988 0             DM_Vx    126.8739   27470.5356  subfind_density()/subfind_density.c/95
  10 2992 0  	  DensDataResult     93.4648   27942.8318  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  29 2990 0             DM_Vz    126.8733   27725.7470  subfind_density()/subfind_density.c/97
  29 2991 0    	  DensDataGet    132.3572   27858.1043  subfind_density()/subfind_density.c/161
  29 2992 0  	  DensDataResult     99.2679   27957.3722  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  98 2989 0             DM_Vy    126.8739   27597.4096  subfind_density()/subfind_density.c/96


  98 2990 0             DM_Vz    126.8739   27724.2835  subfind_density()/subfind_density.c/97

Task=10: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.1682 MB).
task 10: endrun called with an error level of 812



Task=29: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.6278 MB).
task 29: endrun called with an error level of 812


  98 2991 0    	  DensDataGet    115.5894   27839.8728  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 10
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 29
  98 2992 0  	  DensDataResult     86.6920   27926.5649  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=98: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.4351 MB).
task 98: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 98
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 104 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 104 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 104 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 104 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 104 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 104 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 104 2973 0         FOF_PList    380.5594   18861.4116  fof_fof()/fof.c/249
 104 2974 0         FOF_GList    634.2656   19495.6772  fof_fof()/fof.c/337
 104 2975 0             Group     10.7557   19506.4329  fof_fof()/fof.c/380
 104 2976 0   DomainNodeIndex      0.3495   19506.7824  force_treeallocate()/forcetree.c/4800
 104 2977 0        Nodes_base   3737.6006   23244.3831  force_treeallocate()/forcetree.c/4803
 104 2978 0     Extnodes_base   3114.6672   26359.0503  force_treeallocate()/forcetree.c/4811
 104 2979 0          Nextnode    222.3050   26581.3553  force_treeallocate()/forcetree.c/4820
 104 2980 0            Father    221.9056   26803.2609  force_treeallocate()/forcetree.c/4827
 104 2981 0           Ngblist     63.4266   26866.6875  subfind_density()/subfind_density.c/78
 104 2982 0         Dist2list    126.8531   26993.5406  subfind_density()/subfind_density.c/79
 104 2983 0    DataIndexTable     21.8182   27015.3588  subfind_density()/subfind_density.c/87
 104 2984 0      DataNodeList     58.1818   27073.5406  subfind_density()/subfind_density.c/89
 104 2985 0              Left    126.8531   27200.3937  subfind_density()/subfind_density.c/91
 104 2986 0             Right    126.8531   27327.2468  subfind_density()/subfind_density.c/92
 104 2987 0              Todo     15.8566   27343.1035  subfind_density()/subfind_density.c/93
 104 2988 0             DM_Vx    126.8531   27469.9566  subfind_density()/subfind_density.c/95
 104 2989 0             DM_Vy    126.8531   27596.8097  subfind_density()/subfind_density.c/96
 104 2990 0             DM_Vz    126.8531   27723.6629  subfind_density()/subfind_density.c/97
 104 2991 0    	  DensDataGet    117.2621   27840.9249  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 104 2992 0  	  DensDataResult     87.9466   27928.8715  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  27 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  27 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  27 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  27 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  27 2973 0         FOF_PList    380.3824   18861.2346  fof_fof()/fof.c/249
  27 2974 0         FOF_GList    633.9707   19495.2053  fof_fof()/fof.c/337
  27 2975 0             Group     12.1970   19507.4023  fof_fof()/fof.c/380
  27 2976 0   DomainNodeIndex      0.3495   19507.7518  force_treeallocate()/forcetree.c/4800
  27 2977 0        Nodes_base   3737.6006   23245.3524  force_treeallocate()/forcetree.c/4803
  27 2978 0     Extnodes_base   3114.6672   26360.0197  force_treeallocate()/forcetree.c/4811
  27 2979 0          Nextnode    222.3050   26582.3247  force_treeallocate()/forcetree.c/4820

Task=104: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.1285 MB).
task 104: endrun called with an error level of 812


  27 2980 0            Father    221.9056   26804.2303  force_treeallocate()/forcetree.c/4827
  27 2981 0           Ngblist     63.3971   26867.6273  subfind_density()/subfind_density.c/78
  27 2982 0         Dist2list    126.7941   26994.4215  subfind_density()/subfind_density.c/79
  27 2983 0    DataIndexTable     21.8182   27016.2397  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 104
  27 2984 0      DataNodeList     58.1818   27074.4215  subfind_density()/subfind_density.c/89
  27 2985 0              Left    126.7941   27201.2156  subfind_density()/subfind_density.c/91
  27 2986 0             Right    126.7941   27328.0098  subfind_density()/subfind_density.c/92
  27 2987 0              Todo     15.8493   27343.8590  subfind_density()/subfind_density.c/93
 An HDF5 error was detected, we'd better stop.
  27 2988 0             DM_Vx    126.7941   27470.6532  subfind_density()/subfind_density.c/95
  27 2989 0             DM_Vy    126.7941   27597.4473  subfind_density()/subfind_density.c/96
  27 2990 0             DM_Vz    126.7941   27724.2415  subfind_density()/subfind_density.c/97
HDF5-DIAG: Error detected in HDF5 (1.8.12)   27 2991 0    	  DensDataGet    130.7977   27855.0391  subfind_density()/subfind_density.c/161
  27 2992 0  	  DensDataResult     98.0983   27953.1374  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
MPI-process 104:

Task=27: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.8626 MB).
task 27: endrun called with an error level of 812


  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 27
task 104: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 104
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 107 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 107 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 107 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 107 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 107 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 107 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 107 2973 0         FOF_PList    379.4007   18860.2529  fof_fof()/fof.c/249
 107 2974 0         FOF_GList    632.3346   19492.5875  fof_fof()/fof.c/337
 107 2975 0             Group     12.1918   19504.7792  fof_fof()/fof.c/380
 107 2976 0   DomainNodeIndex      0.3495   19505.1288  force_treeallocate()/forcetree.c/4800
 107 2977 0        Nodes_base   3737.6006   23242.7294  force_treeallocate()/forcetree.c/4803
 107 2978 0     Extnodes_base   3114.6672   26357.3966  force_treeallocate()/forcetree.c/4811
 107 2979 0          Nextnode    222.3050   26579.7016  force_treeallocate()/forcetree.c/4820
 107 2980 0            Father    221.9056   26801.6072  force_treeallocate()/forcetree.c/4827
 107 2981 0           Ngblist     63.2335   26864.8407  subfind_density()/subfind_density.c/78
  20 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  20 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  20 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  20 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  20 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  20 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  20 2973 0         FOF_PList    379.7142   18860.5664  fof_fof()/fof.c/249
  20 2974 0         FOF_GList    632.8571   19493.4235  fof_fof()/fof.c/337
  20 2975 0             Group     14.1684   19507.5919  fof_fof()/fof.c/380
  20 2976 0   DomainNodeIndex      0.3495   19507.9414  force_treeallocate()/forcetree.c/4800
 107 2982 0         Dist2list    126.4669   26991.3076  subfind_density()/subfind_density.c/79
 107 2983 0    DataIndexTable     21.8182   27013.1258  subfind_density()/subfind_density.c/87
 107 2984 0      DataNodeList     58.1818   27071.3076  subfind_density()/subfind_density.c/89
 107 2985 0              Left    126.4669   27197.7745  subfind_density()/subfind_density.c/91
  20 2977 0        Nodes_base   3737.6006   23245.5420  force_treeallocate()/forcetree.c/4803
  20 2978 0     Extnodes_base   3114.6672   26360.2092  force_treeallocate()/forcetree.c/4811
 107 2986 0             Right    126.4669   27324.2414  subfind_density()/subfind_density.c/92
 107 2987 0              Todo     15.8084   27340.0498  subfind_density()/subfind_density.c/93
 107 2988 0             DM_Vx    126.4669   27466.5167  subfind_density()/subfind_density.c/95
 107 2989 0             DM_Vy    126.4669   27592.9836  subfind_density()/subfind_density.c/96
 107 2990 0             DM_Vz    126.4669   27719.4505  subfind_density()/subfind_density.c/97
  20 2979 0          Nextnode    222.3050   26582.5143  force_treeallocate()/forcetree.c/4820
  20 2980 0            Father    221.9056   26804.4199  force_treeallocate()/forcetree.c/4827
  20 2981 0           Ngblist     63.2857   26867.7056  subfind_density()/subfind_density.c/78
  20 2982 0         Dist2list    126.5714   26994.2770  subfind_density()/subfind_density.c/79
 107 2991 0    	  DensDataGet    119.2104   27838.6609  subfind_density()/subfind_density.c/161
 107 2992 0  	  DensDataResult     89.4078   27928.0688  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  20 2983 0    DataIndexTable     21.8182   27016.0951  subfind_density()/subfind_density.c/87
  20 2984 0      DataNodeList     58.1818   27074.2769  subfind_density()/subfind_density.c/89
  20 2985 0              Left    126.5714   27200.8484  subfind_density()/subfind_density.c/91

Task=107: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.9312 MB).
task 107: endrun called with an error level of 812


  20 2986 0             Right    126.5714   27327.4198  subfind_density()/subfind_density.c/92
  20 2987 0              Todo     15.8214   27343.2412  subfind_density()/subfind_density.c/93
  20 2988 0             DM_Vx    126.5714   27469.8126  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 107
  20 2989 0             DM_Vy    126.5714   27596.3840  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  20 2990 0             DM_Vz    126.5714   27722.9554  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  12 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  12 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  12 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  12 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  12 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  12 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  12 2973 0         FOF_PList    380.6361   18861.4882  fof_fof()/fof.c/249
  12 2974 0         FOF_GList    634.3934   19495.8817  fof_fof()/fof.c/337
  12 2975 0             Group     11.4509   19507.3326  fof_fof()/fof.c/380
  12 2976 0   DomainNodeIndex      0.3495   19507.6821  force_treeallocate()/forcetree.c/4800
  12 2977 0        Nodes_base   3737.6006   23245.2827  force_treeallocate()/forcetree.c/4803
  12 2978 0     Extnodes_base   3114.6672   26359.9500  force_treeallocate()/forcetree.c/4811
  20 2991 0    	  DensDataGet    117.0504   27840.0058  subfind_density()/subfind_density.c/161
 115 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 115 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 115 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 115 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 115 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 115 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 115 2973 0         FOF_PList    380.4583   18861.3105  fof_fof()/fof.c/249
 115 2974 0         FOF_GList    634.0971   19495.4076  fof_fof()/fof.c/337
 115 2975 0             Group     12.8859   19508.2935  fof_fof()/fof.c/380
 115 2976 0   DomainNodeIndex      0.3495   19508.6430  force_treeallocate()/forcetree.c/4800
 115 2977 0        Nodes_base   3737.6006   23246.2437  force_treeallocate()/forcetree.c/4803
 115 2978 0     Extnodes_base   3114.6672   26360.9109  force_treeallocate()/forcetree.c/4811
 115 2979 0          Nextnode    222.3050   26583.2159  force_treeallocate()/forcetree.c/4820
 115 2980 0            Father    221.9056   26805.1215  force_treeallocate()/forcetree.c/4827
 115 2981 0           Ngblist     63.4097   26868.5312  subfind_density()/subfind_density.c/78
 115 2982 0         Dist2list    126.8194   26995.3507  subfind_density()/subfind_density.c/79
 115 2983 0    DataIndexTable     21.8182   27017.1688  subfind_density()/subfind_density.c/87
 115 2984 0      DataNodeList     58.1818   27075.3506  subfind_density()/subfind_density.c/89
  12 2979 0          Nextnode    222.3050   26582.2550  force_treeallocate()/forcetree.c/4820
  20 2992 0  	  DensDataResult     87.7878   27927.7935  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 115 2985 0              Left    126.8194   27202.1701  subfind_density()/subfind_density.c/91
 115 2986 0             Right    126.8194   27328.9895  subfind_density()/subfind_density.c/92
 115 2987 0              Todo     15.8524   27344.8419  subfind_density()/subfind_density.c/93
 115 2988 0             DM_Vx    126.8194   27471.6613  subfind_density()/subfind_density.c/95
 115 2989 0             DM_Vy    126.8194   27598.4808  subfind_density()/subfind_density.c/96
 115 2990 0             DM_Vz    126.8194   27725.3002  subfind_density()/subfind_density.c/97
  12 2980 0            Father    221.9056   26804.1606  force_treeallocate()/forcetree.c/4827
  12 2981 0           Ngblist     63.4393   26867.5999  subfind_density()/subfind_density.c/78
  12 2982 0         Dist2list    126.8787   26994.4786  subfind_density()/subfind_density.c/79

Task=20: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.2065 MB).
task 20: endrun called with an error level of 812


 115 2991 0    	  DensDataGet    116.3545   27841.6547  subfind_density()/subfind_density.c/161
 115 2992 0  	  DensDataResult     87.2659   27928.9206  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=115: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.0794 MB).
  12 2983 0    DataIndexTable     21.8182   27016.2968  subfind_density()/subfind_density.c/87
  12 2984 0      DataNodeList     58.1818   27074.4786  subfind_density()/subfind_density.c/89
  12 2985 0              Left    126.8787   27201.3573  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 20
task 115: endrun called with an error level of 812


  12 2986 0             Right    126.8787   27328.2359  subfind_density()/subfind_density.c/92
  12 2987 0              Todo     15.8598   27344.0958  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 115
  12 2988 0             DM_Vx    126.8787   27470.9745  subfind_density()/subfind_density.c/95
  12 2989 0             DM_Vy    126.8787   27597.8532  subfind_density()/subfind_density.c/96
  12 2990 0             DM_Vz    126.8787   27724.7318  subfind_density()/subfind_density.c/97
  12 2991 0    	  DensDataGet    132.8337   27857.5656  subfind_density()/subfind_density.c/161
  12 2992 0  	  DensDataResult     99.6253   27957.1909  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=12: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.8091 MB).
task 12: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 12
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 106 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 106 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 106 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 106 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 106 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 106 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
 106 2973 0         FOF_PList    380.2043   18861.0565  fof_fof()/fof.c/249
 106 2974 0         FOF_GList    633.6739   19494.7304  fof_fof()/fof.c/337
 106 2975 0             Group     12.4398   19507.1702  fof_fof()/fof.c/380
 106 2976 0   DomainNodeIndex      0.3495   19507.5197  force_treeallocate()/forcetree.c/4800
 106 2977 0        Nodes_base   3737.6006   23245.1203  force_treeallocate()/forcetree.c/4803
 106 2978 0     Extnodes_base   3114.6672   26359.7875  force_treeallocate()/forcetree.c/4811
 106 2979 0          Nextnode    222.3050   26582.0926  force_treeallocate()/forcetree.c/4820
 106 2980 0            Father    221.9056   26803.9982  force_treeallocate()/forcetree.c/4827
 106 2981 0           Ngblist     63.3674   26867.3656  subfind_density()/subfind_density.c/78
 106 2982 0         Dist2list    126.7348   26994.1003  subfind_density()/subfind_density.c/79
 106 2983 0    DataIndexTable     21.8182   27015.9185  subfind_density()/subfind_density.c/87
 106 2984 0      DataNodeList     58.1818   27074.1003  subfind_density()/subfind_density.c/89
 106 2985 0              Left    126.7348   27200.8351  subfind_density()/subfind_density.c/91
 106 2986 0             Right    126.7348   27327.5699  subfind_density()/subfind_density.c/92
 106 2987 0              Todo     15.8419   27343.4117  subfind_density()/subfind_density.c/93
 106 2988 0             DM_Vx    126.7348   27470.1465  subfind_density()/subfind_density.c/95
 106 2989 0             DM_Vy    126.7348   27596.8813  subfind_density()/subfind_density.c/96
 106 2990 0             DM_Vz    126.7348   27723.6161  subfind_density()/subfind_density.c/97
 106 2991 0    	  DensDataGet    124.1736   27847.7896  subfind_density()/subfind_density.c/161
 106 2992 0  	  DensDataResult     93.1302   27940.9198  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=106: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.0802 MB).
task 106: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 106
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  13 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  13 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  13 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  13 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  13 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  13 2972 0          TopNodes      5.1928   18480.8522  domain_allocate()/domain.c/570
  13 2973 0         FOF_PList    380.5858   18861.4380  fof_fof()/fof.c/249
  13 2974 0         FOF_GList    634.3097   19495.7477  fof_fof()/fof.c/337
  13 2975 0             Group     12.6379   19508.3856  fof_fof()/fof.c/380
  13 2976 0   DomainNodeIndex      0.3495   19508.7351  force_treeallocate()/forcetree.c/4800
  13 2977 0        Nodes_base   3737.6006   23246.3358  force_treeallocate()/forcetree.c/4803
  13 2978 0     Extnodes_base   3114.6672   26361.0030  force_treeallocate()/forcetree.c/4811
  13 2979 0          Nextnode    222.3050   26583.3080  force_treeallocate()/forcetree.c/4820
  13 2980 0            Father    221.9056   26805.2136  force_treeallocate()/forcetree.c/4827
  13 2981 0           Ngblist     63.4310   26868.6446  subfind_density()/subfind_density.c/78
  13 2982 0         Dist2list    126.8619   26995.5065  subfind_density()/subfind_density.c/79
  13 2983 0    DataIndexTable     21.8182   27017.3247  subfind_density()/subfind_density.c/87
  13 2984 0      DataNodeList     58.1818   27075.5065  subfind_density()/subfind_density.c/89
  13 2985 0              Left    126.8619   27202.3684  subfind_density()/subfind_density.c/91
  13 2986 0             Right    126.8619   27329.2304  subfind_density()/subfind_density.c/92
  13 2987 0              Todo     15.8577   27345.0881  subfind_density()/subfind_density.c/93
  13 2988 0             DM_Vx    126.8619   27471.9501  subfind_density()/subfind_density.c/95
  13 2989 0             DM_Vy    126.8619   27598.8120  subfind_density()/subfind_density.c/96
  13 2990 0             DM_Vz    126.8619   27725.6739  subfind_density()/subfind_density.c/97
  13 2991 0    	  DensDataGet    145.5098   27871.1838  subfind_density()/subfind_density.c/161
  13 2992 0  	  DensDataResult    109.1324   27980.3161  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=13: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=19.6839 MB).
task 13: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 13
-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63111.85 Mb (on task=120), Smallest =   62842.60 Mb (on task=0), Average =   63039.98 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1689.17 Mb (on task=0), Smallest =    1420.21 Mb (on task=120), Average =    1492.09 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.11 Mb
AllocMem:	 Largest =    1689.17 Mb (on task=0), Smallest =    1420.21 Mb (on task=120), Average =    1492.09 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_024_z000p155/snap_024_z000p155.0.hdf5' on task=0 (contains 134325771 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  133957536  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      9657  (tot=     0000134208) masstab=0
Type 3 (bulge):   358578  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.876876

Setting next time for line of sight file to Time_next= 0.869993

Setting next time for snipshot file to Time_next= 0.874207

Setting next time for fof output to Time_next= 0.874207
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=8.4851 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=102633
After=103401
NTopleaves= 90476  NTopnodes=103401 (space for 465370)
gravity work-load balance=1.0004   memory-balance=1.00039   SPH work-load balance=0
iter=0 exchange of 2108994229 particles (ret=0)
domain decomposition done. (took 37.4801 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5696 MByte in top-level domain structure

Allocated 7296.25 MByte for BH-tree, and 0.345139 Mbyte for top-leaves.  (presently allocted 25777.4 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.3 Mbyte (on task=73), Smallest = 26119.7 Mbyte, Average = 26120.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  73 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  73 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  73 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  73 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  73 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  73 2972 0          TopNodes      5.1278   18480.7871  domain_allocate()/domain.c/570
  73 2973 0   DomainNodeIndex      0.3451   18481.1322  force_treeallocate()/forcetree.c/4800
  73 2974 0        Nodes_base   3737.4805   22218.6128  force_treeallocate()/forcetree.c/4803
  73 2975 0     Extnodes_base   3114.5671   25333.1799  force_treeallocate()/forcetree.c/4811
  73 2976 0          Nextnode    222.3000   25555.4799  force_treeallocate()/forcetree.c/4820
  73 2977 0            Father    221.9056   25777.3855  force_treeallocate()/forcetree.c/4827
  73 2978 0           Ngblist     63.4266   25840.8121  density()/density.c/690
  73 2979 0              Left    126.8532   25967.6653  density()/density.c/692
  73 2980 0             Right    126.8532   26094.5185  density()/density.c/693
  73 2981 0    DataIndexTable      7.0175   26101.5360  density()/density.c/738
  73 2982 0      DataNodeList     18.7134   26120.2495  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.101.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.155.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.4 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=94017
After=103937
NTopleaves= 90945  NTopnodes=103937 (space for 465370)
gravity work-load balance=1.00047   memory-balance=1.00047   SPH work-load balance=0
iter=0 exchange of 2108120493 particles (ret=0)
domain decomposition done. (took 31.8091 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5451 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.8 MB)
Tree construction.

Start linking particles (presently allocated=26538.3 MB)
links on local processor done (took 173.815 sec).
Marked=0228207724 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26862.4 MB) 
have done 0002883665 cross links (processed 0228207724, took 26.4023 sec)
have done 0001279811 cross links (processed 0190759859, took 25.5912 sec)
have done 0000920995 cross links (processed 0166923232, took 23.709 sec)
have done 0000618361 cross links (processed 0119657552, took 22.7317 sec)
have done 0000400685 cross links (processed 0076443830, took 20.4842 sec)
have done 0000260649 cross links (processed 0041133205, took 13.0847 sec)
have done 0000172961 cross links (processed 0027287474, took 5.60515 sec)
have done 0000110373 cross links (processed 0027324735, took 14.7754 sec)
have done 0000069129 cross links (processed 0014771243, took 7.83429 sec)
have done 0000039119 cross links (processed 0005237183, took 9.89294 sec)
have done 0000026938 cross links (processed 0001062840, took 1.03558 sec)
have done 0000020166 cross links (processed 0000610086, took 0.558422 sec)
have done 0000011437 cross links (processed 0000272230, took 0.251363 sec)
have done 0000004497 cross links (processed 0000082892, took 0.208347 sec)
have done 0000001655 cross links (processed 0000029050, took 0.184758 sec)
have done 0000000651 cross links (processed 0000019267, took 0.193143 sec)
have done 0000000288 cross links (processed 0000013632, took 0.197114 sec)
have done 0000000145 cross links (processed 0000004624, took 0.178117 sec)
have done 0000000077 cross links (processed 0000002139, took 0.180911 sec)
have done 0000000027 cross links (processed 0000000466, took 0.178169 sec)
have done 0000000010 cross links (processed 0000000123, took 0.179518 sec)
have done 0000000004 cross links (processed 0000000026, took 0.177407 sec)
have done 0000000001 cross links (processed 0000000009, took 0.173699 sec)
have done 0000000000 cross links (processed 0000000002, took 0.173895 sec)
Local groups found.

group finding took = 348.412 sec
Start finding nearest dm-particle (presently allocated=26538.3 MB)
still finding nearest... (presently allocated=26805.7 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.435648 sec
compiling local group data and catalogue took = 7.59465 sec

Total number of groups with at least 32 particles: 1284773
Largest group has 137266522 particles.
Total number of particles in groups: 1293579451

group properties are now allocated.. (presently allocated=19506.5 MB)
computation of group properties took = 0.865165 sec
start global sorting of group catalogues number 24
LargestGroup = 107004.606943; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.222518 sec
Group catalogues globally sorted. took = 10.3327 sec
starting saving of group catalogue number 24
data//groups_024_z000p155
Group catalogues number 24 saved. took = 16.8917 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 12.3103 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 2.05734 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 7.16422 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2965 0     ProcessedFlag     55.4764      57.4321  allocate_memory()/allocate.c/52
   0 2966 0  NextActiveParticle    221.9056     279.3377  allocate_memory()/allocate.c/55
   0 2967 0     NextInTimeBin    221.9056     501.2433  allocate_memory()/allocate.c/58
   0 2968 0     PrevInTimeBin    221.9056     723.1489  allocate_memory()/allocate.c/61
   0 2969 0                 P  17752.4469   18475.5958  allocate_memory()/allocate.c/67
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2972 0          TopNodes      5.1544   18480.8126  domain_allocate()/domain.c/570
   0 2973 0         FOF_PList    380.4206   18861.2332  fof_fof()/fof.c/249
   0 2974 0         FOF_GList    634.0343   19495.2675  fof_fof()/fof.c/337
   0 2975 0             Group     11.2361   19506.5037  fof_fof()/fof.c/380
   0 2976 0   DomainNodeIndex      0.3469   19506.8506  force_treeallocate()/forcetree.c/4800
   0 2977 0        Nodes_base   3737.5296   23244.3802  force_treeallocate()/forcetree.c/4803
   0 2978 0     Extnodes_base   3114.6080   26358.9882  force_treeallocate()/forcetree.c/4811
   0 2979 0          Nextnode    222.3021   26581.2903  force_treeallocate()/forcetree.c/4820
   0 2980 0            Father    221.9056   26803.1959  force_treeallocate()/forcetree.c/4827
 127 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 127 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 127 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 127 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 127 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 127 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 127 2973 0         FOF_PList    380.5565   18861.3702  fof_fof()/fof.c/249
 127 2974 0         FOF_GList    634.2609   19495.6311  fof_fof()/fof.c/337
 127 2975 0             Group     13.0955   19508.7266  fof_fof()/fof.c/380
 127 2976 0   DomainNodeIndex      0.3469   19509.0735  force_treeallocate()/forcetree.c/4800
 127 2977 0        Nodes_base   3737.5296   23246.6031  force_treeallocate()/forcetree.c/4803
 127 2978 0     Extnodes_base   3114.6080   26361.2111  force_treeallocate()/forcetree.c/4811
 127 2979 0          Nextnode    222.3021   26583.5132  force_treeallocate()/forcetree.c/4820
 127 2980 0            Father    221.9056   26805.4188  force_treeallocate()/forcetree.c/4827
 127 2981 0           Ngblist     63.4261   26868.8449  subfind_density()/subfind_density.c/78
 127 2982 0         Dist2list    126.8522   26995.6971  subfind_density()/subfind_density.c/79
 127 2983 0    DataIndexTable     21.8182   27017.5152  subfind_density()/subfind_density.c/87
   0 2981 0           Ngblist     63.4034   26866.5993  subfind_density()/subfind_density.c/78
   0 2982 0         Dist2list    126.8069   26993.4062  subfind_density()/subfind_density.c/79
   0 2983 0    DataIndexTable     21.8182   27015.2244  subfind_density()/subfind_density.c/87
 127 2984 0      DataNodeList     58.1818   27075.6970  subfind_density()/subfind_density.c/89
 127 2985 0              Left    126.8522   27202.5492  subfind_density()/subfind_density.c/91
 127 2986 0             Right    126.8522   27329.4014  subfind_density()/subfind_density.c/92
 127 2987 0              Todo     15.8565   27345.2579  subfind_density()/subfind_density.c/93
 127 2988 0             DM_Vx    126.8522   27472.1101  subfind_density()/subfind_density.c/95
 127 2989 0             DM_Vy    126.8522   27598.9622  subfind_density()/subfind_density.c/96
   0 2984 0      DataNodeList     58.1818   27073.4062  subfind_density()/subfind_density.c/89
   0 2985 0              Left    126.8069   27200.2130  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 127 2990 0             DM_Vz    126.8522   27725.8144  subfind_density()/subfind_density.c/97
 127 2991 0    	  DensDataGet    137.8253   27863.6397  subfind_density()/subfind_density.c/161
 127 2992 0  	  DensDataResult    103.3689   27967.0086  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2986 0             Right    126.8069   27327.0199  subfind_density()/subfind_density.c/92
   0 2987 0              Todo     15.8509   27342.8707  subfind_density()/subfind_density.c/93
 122 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 122 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 122 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 122 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 122 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 122 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 122 2973 0         FOF_PList    380.5773   18861.3910  fof_fof()/fof.c/249
 122 2974 0         FOF_GList    634.2955   19495.6866  fof_fof()/fof.c/337
 122 2975 0             Group     11.2922   19506.9787  fof_fof()/fof.c/380
 122 2976 0   DomainNodeIndex      0.3469   19507.3257  force_treeallocate()/forcetree.c/4800
 122 2977 0        Nodes_base   3737.5296   23244.8553  force_treeallocate()/forcetree.c/4803
 122 2978 0     Extnodes_base   3114.6080   26359.4633  force_treeallocate()/forcetree.c/4811
 121 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 121 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 121 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 121 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 121 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 121 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 121 2973 0         FOF_PList    380.4981   18861.3118  fof_fof()/fof.c/249
 121 2974 0         FOF_GList    634.1636   19495.4754  fof_fof()/fof.c/337
 121 2975 0             Group     11.7352   19507.2106  fof_fof()/fof.c/380
 121 2976 0   DomainNodeIndex      0.3469   19507.5575  force_treeallocate()/forcetree.c/4800
 121 2977 0        Nodes_base   3737.5296   23245.0871  force_treeallocate()/forcetree.c/4803
 121 2978 0     Extnodes_base   3114.6080   26359.6951  force_treeallocate()/forcetree.c/4811
  14 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  14 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  14 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  14 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  14 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   5 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   5 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   5 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   5 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   5 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   5 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   5 2973 0         FOF_PList    380.5776   18861.3912  fof_fof()/fof.c/249
   5 2974 0         FOF_GList    634.2959   19495.6872  fof_fof()/fof.c/337
   5 2975 0             Group     12.2239   19507.9111  fof_fof()/fof.c/380
   5 2976 0   DomainNodeIndex      0.3469   19508.2580  force_treeallocate()/forcetree.c/4800
   5 2977 0        Nodes_base   3737.5296   23245.7876  force_treeallocate()/forcetree.c/4803
   5 2978 0     Extnodes_base   3114.6080   26360.3956  force_treeallocate()/forcetree.c/4811
   5 2979 0          Nextnode    222.3021   26582.6977  force_treeallocate()/forcetree.c/4820
  10 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  10 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  10 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  10 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  10 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  10 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  10 2973 0         FOF_PList    380.4203   18861.2340  fof_fof()/fof.c/249
  10 2974 0         FOF_GList    634.0339   19495.2679  fof_fof()/fof.c/337
  10 2975 0             Group     11.9625   19507.2303  fof_fof()/fof.c/380
  10 2976 0   DomainNodeIndex      0.3469   19507.5772  force_treeallocate()/forcetree.c/4800
  10 2977 0        Nodes_base   3737.5296   23245.1068  force_treeallocate()/forcetree.c/4803

Task=127: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=32.9914 MB).
task 127: endrun called with an error level of 812


   6 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   6 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   6 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   6 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   6 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   6 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   6 2973 0         FOF_PList    380.2835   18861.0972  fof_fof()/fof.c/249
   6 2974 0         FOF_GList    633.8058   19494.9030  fof_fof()/fof.c/337
   6 2975 0             Group     12.6577   19507.5607  fof_fof()/fof.c/380
   6 2976 0   DomainNodeIndex      0.3469   19507.9076  force_treeallocate()/forcetree.c/4800
 125 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 125 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 125 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 125 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 125 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 125 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 125 2973 0         FOF_PList    380.5142   18861.3279  fof_fof()/fof.c/249
 125 2974 0         FOF_GList    634.1904   19495.5183  fof_fof()/fof.c/337
 125 2975 0             Group     10.4175   19505.9358  fof_fof()/fof.c/380
 125 2976 0   DomainNodeIndex      0.3469   19506.2827  force_treeallocate()/forcetree.c/4800
 125 2977 0        Nodes_base   3737.5296   23243.8123  force_treeallocate()/forcetree.c/4803
 125 2978 0     Extnodes_base   3114.6080   26358.4203  force_treeallocate()/forcetree.c/4811
 125 2979 0          Nextnode    222.3021   26580.7224  force_treeallocate()/forcetree.c/4820
 125 2980 0            Father    221.9056   26802.6280  force_treeallocate()/forcetree.c/4827
 125 2981 0           Ngblist     63.4190   26866.0470  subfind_density()/subfind_density.c/78
   2 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   2 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   2 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   2 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   2 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   2 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   2 2973 0         FOF_PList    380.2533   18861.0670  fof_fof()/fof.c/249
   2 2974 0         FOF_GList    633.7556   19494.8226  fof_fof()/fof.c/337
   2 2975 0             Group     12.0268   19506.8494  fof_fof()/fof.c/380
   2 2976 0   DomainNodeIndex      0.3469   19507.1963  force_treeallocate()/forcetree.c/4800
   2 2977 0        Nodes_base   3737.5296   23244.7259  force_treeallocate()/forcetree.c/4803
   2 2978 0     Extnodes_base   3114.6080   26359.3339  force_treeallocate()/forcetree.c/4811
   2 2979 0          Nextnode    222.3021   26581.6360  force_treeallocate()/forcetree.c/4820
   2 2980 0            Father    221.9056   26803.5416  force_treeallocate()/forcetree.c/4827
   2 2981 0           Ngblist     63.3756   26866.9172  subfind_density()/subfind_density.c/78
 117 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 117 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 117 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 117 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 117 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 117 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 117 2973 0         FOF_PList    380.5097   18861.3234  fof_fof()/fof.c/249
 117 2974 0         FOF_GList    634.1829   19495.5063  fof_fof()/fof.c/337
 117 2975 0             Group     12.4844   19507.9907  fof_fof()/fof.c/380
 117 2976 0   DomainNodeIndex      0.3469   19508.3376  force_treeallocate()/forcetree.c/4800
 117 2977 0        Nodes_base   3737.5296   23245.8672  force_treeallocate()/forcetree.c/4803
 117 2978 0     Extnodes_base   3114.6080   26360.4752  force_treeallocate()/forcetree.c/4811
 117 2979 0          Nextnode    222.3021   26582.7773  force_treeallocate()/forcetree.c/4820
 117 2980 0            Father    221.9056   26804.6829  force_treeallocate()/forcetree.c/4827
 118 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   0 2988 0             DM_Vx    126.8069   27469.6776  subfind_density()/subfind_density.c/95
   0 2989 0             DM_Vy    126.8069   27596.4845  subfind_density()/subfind_density.c/96
 122 2979 0          Nextnode    222.3021   26581.7653  force_treeallocate()/forcetree.c/4820
 122 2980 0            Father    221.9056   26803.6709  force_treeallocate()/forcetree.c/4827
 122 2981 0           Ngblist     63.4296   26867.1005  subfind_density()/subfind_density.c/78
 121 2979 0          Nextnode    222.3021   26581.9972  force_treeallocate()/forcetree.c/4820
 121 2980 0            Father    221.9056   26803.9028  force_treeallocate()/forcetree.c/4827
 121 2981 0           Ngblist     63.4164   26867.3192  subfind_density()/subfind_density.c/78
 121 2982 0         Dist2list    126.8327   26994.1519  subfind_density()/subfind_density.c/79
  14 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  14 2973 0         FOF_PList    380.5549   18861.3686  fof_fof()/fof.c/249
  14 2974 0         FOF_GList    634.2581   19495.6267  fof_fof()/fof.c/337
   5 2980 0            Father    221.9056   26804.6033  force_treeallocate()/forcetree.c/4827
   5 2981 0           Ngblist     63.4296   26868.0329  subfind_density()/subfind_density.c/78
   5 2982 0         Dist2list    126.8592   26994.8921  subfind_density()/subfind_density.c/79
   5 2983 0    DataIndexTable     21.8182   27016.7103  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  10 2978 0     Extnodes_base   3114.6080   26359.7149  force_treeallocate()/forcetree.c/4811
  10 2979 0          Nextnode    222.3021   26582.0169  force_treeallocate()/forcetree.c/4820
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 127
   6 2977 0        Nodes_base   3737.5296   23245.4372  force_treeallocate()/forcetree.c/4803
   6 2978 0     Extnodes_base   3114.6080   26360.0452  force_treeallocate()/forcetree.c/4811
   6 2979 0          Nextnode    222.3021   26582.3473  force_treeallocate()/forcetree.c/4820
 125 2982 0         Dist2list    126.8381   26992.8851  subfind_density()/subfind_density.c/79
 125 2983 0    DataIndexTable     21.8182   27014.7032  subfind_density()/subfind_density.c/87
 125 2984 0      DataNodeList     58.1818   27072.8850  subfind_density()/subfind_density.c/89
 125 2985 0              Left    126.8381   27199.7231  subfind_density()/subfind_density.c/91
 125 2986 0             Right    126.8381   27326.5612  subfind_density()/subfind_density.c/92
   2 2982 0         Dist2list    126.7511   26993.6683  subfind_density()/subfind_density.c/79
   2 2983 0    DataIndexTable     21.8182   27015.4864  subfind_density()/subfind_density.c/87
   2 2984 0      DataNodeList     58.1818   27073.6682  subfind_density()/subfind_density.c/89
 117 2981 0           Ngblist     63.4183   26868.1012  subfind_density()/subfind_density.c/78
 117 2982 0         Dist2list    126.8366   26994.9378  subfind_density()/subfind_density.c/79
 117 2983 0    DataIndexTable     21.8182   27016.7559  subfind_density()/subfind_density.c/87
 117 2984 0      DataNodeList     58.1818   27074.9377  subfind_density()/subfind_density.c/89
 117 2985 0              Left    126.8366   27201.7743  subfind_density()/subfind_density.c/91
 118 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 118 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 118 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   0 2990 0             DM_Vz    126.8069   27723.2913  subfind_density()/subfind_density.c/97
   0 2991 0    	  DensDataGet    110.2603   27833.5516  subfind_density()/subfind_density.c/161
 122 2982 0         Dist2list    126.8591   26993.9596  subfind_density()/subfind_density.c/79
 122 2983 0    DataIndexTable     21.8182   27015.7778  subfind_density()/subfind_density.c/87
 122 2984 0      DataNodeList     58.1818   27073.9596  subfind_density()/subfind_density.c/89
 121 2983 0    DataIndexTable     21.8182   27015.9700  subfind_density()/subfind_density.c/87
 121 2984 0      DataNodeList     58.1818   27074.1518  subfind_density()/subfind_density.c/89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  14 2975 0             Group     13.1038   19508.7305  fof_fof()/fof.c/380
  14 2976 0   DomainNodeIndex      0.3469   19509.0774  force_treeallocate()/forcetree.c/4800
  14 2977 0        Nodes_base   3737.5296   23246.6070  force_treeallocate()/forcetree.c/4803
  14 2978 0     Extnodes_base   3114.6080   26361.2150  force_treeallocate()/forcetree.c/4811
  14 2979 0          Nextnode    222.3021   26583.5171  force_treeallocate()/forcetree.c/4820
   5 2984 0      DataNodeList     58.1818   27074.8921  subfind_density()/subfind_density.c/89
   5 2985 0              Left    126.8592   27201.7512  subfind_density()/subfind_density.c/91
   5 2986 0             Right    126.8592   27328.6104  subfind_density()/subfind_density.c/92
   5 2987 0              Todo     15.8574   27344.4678  subfind_density()/subfind_density.c/93
  26 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  26 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  26 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  26 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  26 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  26 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  26 2973 0         FOF_PList    380.5348   18861.3485  fof_fof()/fof.c/249
  26 2974 0         FOF_GList    634.2247   19495.5732  fof_fof()/fof.c/337
  26 2975 0             Group     13.5770   19509.1501  fof_fof()/fof.c/380
  26 2976 0   DomainNodeIndex      0.3469   19509.4970  force_treeallocate()/forcetree.c/4800
  26 2977 0        Nodes_base   3737.5296   23247.0266  force_treeallocate()/forcetree.c/4803
  10 2980 0            Father    221.9056   26803.9225  force_treeallocate()/forcetree.c/4827
  10 2981 0           Ngblist     63.4034   26867.3259  subfind_density()/subfind_density.c/78
  10 2982 0         Dist2list    126.8068   26994.1327  subfind_density()/subfind_density.c/79
  10 2983 0    DataIndexTable     21.8182   27015.9509  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   6 2980 0            Father    221.9056   26804.2529  force_treeallocate()/forcetree.c/4827
   6 2981 0           Ngblist     63.3806   26867.6335  subfind_density()/subfind_density.c/78
   6 2982 0         Dist2list    126.7612   26994.3947  subfind_density()/subfind_density.c/79
 125 2987 0              Todo     15.8548   27342.4159  subfind_density()/subfind_density.c/93
 125 2988 0             DM_Vx    126.8381   27469.2540  subfind_density()/subfind_density.c/95
 125 2989 0             DM_Vy    126.8381   27596.0921  subfind_density()/subfind_density.c/96
   2 2985 0              Left    126.7511   27200.4193  subfind_density()/subfind_density.c/91
   2 2986 0             Right    126.7511   27327.1705  subfind_density()/subfind_density.c/92
   2 2987 0              Todo     15.8439   27343.0144  subfind_density()/subfind_density.c/93
 117 2986 0             Right    126.8366   27328.6109  subfind_density()/subfind_density.c/92
 117 2987 0              Todo     15.8546   27344.4655  subfind_density()/subfind_density.c/93
 117 2988 0             DM_Vx    126.8366   27471.3020  subfind_density()/subfind_density.c/95
 118 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 118 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   0 2992 0  	  DensDataResult     82.6952   27916.2468  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 122 2985 0              Left    126.8591   27200.8187  subfind_density()/subfind_density.c/91
 122 2986 0             Right    126.8591   27327.6778  subfind_density()/subfind_density.c/92
 122 2987 0              Todo     15.8574   27343.5352  subfind_density()/subfind_density.c/93
 121 2985 0              Left    126.8327   27200.9846  subfind_density()/subfind_density.c/91
 121 2986 0             Right    126.8327   27327.8173  subfind_density()/subfind_density.c/92
 121 2987 0              Todo     15.8541   27343.6714  subfind_density()/subfind_density.c/93
 101 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 101 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 101 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 101 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 101 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 101 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 101 2973 0         FOF_PList    380.5806   18861.3943  fof_fof()/fof.c/249
 101 2974 0         FOF_GList    634.3010   19495.6953  fof_fof()/fof.c/337
 101 2975 0             Group     10.4154   19506.1107  fof_fof()/fof.c/380
 101 2976 0   DomainNodeIndex      0.3469   19506.4577  force_treeallocate()/forcetree.c/4800
 101 2977 0        Nodes_base   3737.5296   23243.9873  force_treeallocate()/forcetree.c/4803
 101 2978 0     Extnodes_base   3114.6080   26358.5953  force_treeallocate()/forcetree.c/4811
 101 2979 0          Nextnode    222.3021   26580.8974  force_treeallocate()/forcetree.c/4820
 101 2980 0            Father    221.9056   26802.8029  force_treeallocate()/forcetree.c/4827
  14 2980 0            Father    221.9056   26805.4227  force_treeallocate()/forcetree.c/4827
  14 2981 0           Ngblist     63.4258   26868.8485  subfind_density()/subfind_density.c/78
  14 2982 0         Dist2list    126.8516   26995.7001  subfind_density()/subfind_density.c/79
   5 2988 0             DM_Vx    126.8592   27471.3270  subfind_density()/subfind_density.c/95
   5 2989 0             DM_Vy    126.8592   27598.1862  subfind_density()/subfind_density.c/96
   5 2990 0             DM_Vz    126.8592   27725.0454  subfind_density()/subfind_density.c/97
  26 2978 0     Extnodes_base   3114.6080   26361.6347  force_treeallocate()/forcetree.c/4811
  26 2979 0          Nextnode    222.3021   26583.9367  force_treeallocate()/forcetree.c/4820
  26 2980 0            Father    221.9056   26805.8423  force_treeallocate()/forcetree.c/4827
  26 2981 0           Ngblist     63.4225   26869.2648  subfind_density()/subfind_density.c/78
  26 2982 0         Dist2list    126.8449   26996.1097  subfind_density()/subfind_density.c/79
  10 2984 0      DataNodeList     58.1818   27074.1326  subfind_density()/subfind_density.c/89
  10 2985 0              Left    126.8068   27200.9394  subfind_density()/subfind_density.c/91
  10 2986 0             Right    126.8068   27327.7462  subfind_density()/subfind_density.c/92
  10 2987 0              Todo     15.8509   27343.5970  subfind_density()/subfind_density.c/93
  10 2988 0             DM_Vx    126.8068   27470.4038  subfind_density()/subfind_density.c/95
 126 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 126 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 126 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 126 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 126 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 126 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   6 2983 0    DataIndexTable     21.8182   27016.2128  subfind_density()/subfind_density.c/87
   6 2984 0      DataNodeList     58.1818   27074.3946  subfind_density()/subfind_density.c/89
   6 2985 0              Left    126.7612   27201.1558  subfind_density()/subfind_density.c/91
 125 2990 0             DM_Vz    126.8381   27722.9302  subfind_density()/subfind_density.c/97
 125 2991 0    	  DensDataGet    129.7662   27852.6963  subfind_density()/subfind_density.c/161
 125 2992 0  	  DensDataResult     97.3246   27950.0210  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   2 2988 0             DM_Vx    126.7511   27469.7655  subfind_density()/subfind_density.c/95
   2 2989 0             DM_Vy    126.7511   27596.5166  subfind_density()/subfind_density.c/96
   2 2990 0             DM_Vz    126.7511   27723.2677  subfind_density()/subfind_density.c/97
 117 2989 0             DM_Vy    126.8366   27598.1386  subfind_density()/subfind_density.c/96
 117 2990 0             DM_Vz    126.8366   27724.9752  subfind_density()/subfind_density.c/97
 117 2991 0    	  DensDataGet    118.9434   27843.9186  subfind_density()/subfind_density.c/161
 118 2973 0         FOF_PList    380.5884   18861.4021  fof_fof()/fof.c/249
 118 2974 0         FOF_GList    634.3140   19495.7160  fof_fof()/fof.c/337

Task=0: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.7532 MB).
task 0: endrun called with an error level of 812


 122 2988 0             DM_Vx    126.8591   27470.3943  subfind_density()/subfind_density.c/95
 122 2989 0             DM_Vy    126.8591   27597.2534  subfind_density()/subfind_density.c/96
 122 2990 0             DM_Vz    126.8591   27724.1125  subfind_density()/subfind_density.c/97
 121 2988 0             DM_Vx    126.8327   27470.5041  subfind_density()/subfind_density.c/95
 121 2989 0             DM_Vy    126.8327   27597.3368  subfind_density()/subfind_density.c/96
 101 2981 0           Ngblist     63.4301   26866.2331  subfind_density()/subfind_density.c/78
 101 2982 0         Dist2list    126.8602   26993.0933  subfind_density()/subfind_density.c/79
 101 2983 0    DataIndexTable     21.8182   27014.9114  subfind_density()/subfind_density.c/87
 101 2984 0      DataNodeList     58.1818   27073.0932  subfind_density()/subfind_density.c/89
 101 2985 0              Left    126.8602   27199.9534  subfind_density()/subfind_density.c/91
  14 2983 0    DataIndexTable     21.8182   27017.5183  subfind_density()/subfind_density.c/87
  14 2984 0      DataNodeList     58.1818   27075.7001  subfind_density()/subfind_density.c/89
   5 2991 0    	  DensDataGet    108.2466   27833.2920  subfind_density()/subfind_density.c/161
   5 2992 0  	  DensDataResult     81.1849   27914.4769  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=5: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.5231 MB).
  26 2983 0    DataIndexTable     21.8182   27017.9279  subfind_density()/subfind_density.c/87
  26 2984 0      DataNodeList     58.1818   27076.1097  subfind_density()/subfind_density.c/89
  26 2985 0              Left    126.8449   27202.9546  subfind_density()/subfind_density.c/91
  10 2989 0             DM_Vy    126.8068   27597.2106  subfind_density()/subfind_density.c/96
  10 2990 0             DM_Vz    126.8068   27724.0173  subfind_density()/subfind_density.c/97
  10 2991 0    	  DensDataGet    111.3987   27835.4160  subfind_density()/subfind_density.c/161
 126 2973 0         FOF_PList    380.5798   18861.3935  fof_fof()/fof.c/249
 126 2974 0         FOF_GList    634.2997   19495.6932  fof_fof()/fof.c/337
 126 2975 0             Group     12.0195   19507.7127  fof_fof()/fof.c/380
 126 2976 0   DomainNodeIndex      0.3469   19508.0597  force_treeallocate()/forcetree.c/4800
   6 2986 0             Right    126.7612   27327.9170  subfind_density()/subfind_density.c/92
   6 2987 0              Todo     15.8451   27343.7621  subfind_density()/subfind_density.c/93
   6 2988 0             DM_Vx    126.7612   27470.5233  subfind_density()/subfind_density.c/95

Task=125: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.979 MB).
task 125: endrun called with an error level of 812


   2 2991 0    	  DensDataGet    111.3574   27834.6251  subfind_density()/subfind_density.c/161
   2 2992 0  	  DensDataResult     83.5180   27918.1431  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 117 2992 0  	  DensDataResult     89.2076   27933.1262  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=117: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.8738 MB).
task 117: endrun called with an error level of 812


 118 2975 0             Group     12.9980   19508.7140  fof_fof()/fof.c/380
 118 2976 0   DomainNodeIndex      0.3469   19509.0610  force_treeallocate()/forcetree.c/4800
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 0
 122 2991 0    	  DensDataGet    120.8359   27844.9484  subfind_density()/subfind_density.c/161
 121 2990 0             DM_Vz    126.8327   27724.1695  subfind_density()/subfind_density.c/97
 101 2986 0             Right    126.8602   27326.8136  subfind_density()/subfind_density.c/92
 101 2987 0              Todo     15.8575   27342.6712  subfind_density()/subfind_density.c/93
 101 2988 0             DM_Vx    126.8602   27469.5314  subfind_density()/subfind_density.c/95
  14 2985 0              Left    126.8516   27202.5517  subfind_density()/subfind_density.c/91
task 5: endrun called with an error level of 812


  26 2986 0             Right    126.8449   27329.7996  subfind_density()/subfind_density.c/92
  26 2987 0              Todo     15.8556   27345.6552  subfind_density()/subfind_density.c/93
  26 2988 0             DM_Vx    126.8449   27472.5001  subfind_density()/subfind_density.c/95
  10 2992 0  	  DensDataResult     83.5490   27918.9650  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 126 2977 0        Nodes_base   3737.5296   23245.5893  force_treeallocate()/forcetree.c/4803
 126 2978 0     Extnodes_base   3114.6080   26360.1973  force_treeallocate()/forcetree.c/4811
   6 2989 0             DM_Vy    126.7612   27597.2844  subfind_density()/subfind_density.c/96
   6 2990 0             DM_Vz    126.7612   27724.0456  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 125

Task=2: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.8569 MB).
task 2: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 117
 118 2977 0        Nodes_base   3737.5296   23246.5906  force_treeallocate()/forcetree.c/4803
 An HDF5 error was detected, we'd better stop.
 122 2992 0  	  DensDataResult     90.6269   27935.5753  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------- 121 2991 0    	  DensDataGet    122.3220   27846.4915  subfind_density()/subfind_density.c/161
 101 2989 0             DM_Vy    126.8602   27596.3916  subfind_density()/subfind_density.c/96
 101 2990 0             DM_Vz    126.8602   27723.2518  subfind_density()/subfind_density.c/97
 101 2991 0    	  DensDataGet    131.3660   27854.6178  subfind_density()/subfind_density.c/161
  14 2986 0             Right    126.8516   27329.4034  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 5
  26 2989 0             DM_Vy    126.8449   27599.3450  subfind_density()/subfind_density.c/96
  26 2990 0             DM_Vz    126.8449   27726.1900  subfind_density()/subfind_density.c/97
  26 2991 0    	  DensDataGet    108.9362   27835.1261  subfind_density()/subfind_density.c/161

Task=10: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.035 MB).
task 10: endrun called with an error level of 812


 126 2979 0          Nextnode    222.3021   26582.4994  force_treeallocate()/forcetree.c/4820
   6 2991 0    	  DensDataGet    127.7811   27851.8267  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 2
 118 2978 0     Extnodes_base   3114.6080   26361.1986  force_treeallocate()/forcetree.c/4811
HDF5-DIAG: Error detected in HDF5 (1.8.12) 
 121 2992 0  	  DensDataResult     91.7415   27938.2330  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 101 2992 0  	  DensDataResult     98.5245   27953.1423  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=101: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.8577 MB).
  14 2987 0              Todo     15.8565   27345.2598  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  26 2992 0  	  DensDataResult     81.7021   27916.8282  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 10
 126 2980 0            Father    221.9056   26804.4049  force_treeallocate()/forcetree.c/4827
   6 2992 0  	  DensDataResult     95.8358   27947.6625  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 118 2979 0          Nextnode    222.3021   26583.5006  force_treeallocate()/forcetree.c/4820
MPI-process 0:

Task=122: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.4247 MB).

Task=121: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.767 MB).
task 121: endrun called with an error level of 812


task 101: endrun called with an error level of 812


  14 2988 0             DM_Vx    126.8516   27472.1114  subfind_density()/subfind_density.c/95
   4 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   4 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   4 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   4 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61

Task=26: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.1718 MB).
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 126 2981 0           Ngblist     63.4300   26867.8349  subfind_density()/subfind_density.c/78

Task=6: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.3375 MB).
task 6: endrun called with an error level of 812


   3 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   3 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   3 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   3 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   3 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   3 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   3 2973 0         FOF_PList    380.3965   18861.2102  fof_fof()/fof.c/249
   3 2974 0         FOF_GList    633.9941   19495.2043  fof_fof()/fof.c/337
   3 2975 0             Group     11.3243   19506.5286  fof_fof()/fof.c/380
   3 2976 0   DomainNodeIndex      0.3469   19506.8755  force_treeallocate()/forcetree.c/4800
   3 2977 0        Nodes_base   3737.5296   23244.4051  force_treeallocate()/forcetree.c/4803
   3 2978 0     Extnodes_base   3114.6080   26359.0131  force_treeallocate()/forcetree.c/4811
   3 2979 0          Nextnode    222.3021   26581.3152  force_treeallocate()/forcetree.c/4820
   3 2980 0            Father    221.9056   26803.2208  force_treeallocate()/forcetree.c/4827
 118 2980 0            Father    221.9056   26805.4062  force_treeallocate()/forcetree.c/4827
task 0: endrun called with an error level of 124


task 122: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 121
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 101
  14 2989 0             DM_Vy    126.8516   27598.9631  subfind_density()/subfind_density.c/96
   4 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   4 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   4 2973 0         FOF_PList    380.4381   18861.2518  fof_fof()/fof.c/249
task 26: endrun called with an error level of 812


  11 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  11 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  11 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  11 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  11 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  11 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  11 2973 0         FOF_PList    380.5729   18861.3866  fof_fof()/fof.c/249
  11 2974 0         FOF_GList    634.2882   19495.6748  fof_fof()/fof.c/337
  11 2975 0             Group     10.8118   19506.4866  fof_fof()/fof.c/380
  11 2976 0   DomainNodeIndex      0.3469   19506.8335  force_treeallocate()/forcetree.c/4800
  11 2977 0        Nodes_base   3737.5296   23244.3631  force_treeallocate()/forcetree.c/4803
  11 2978 0     Extnodes_base   3114.6080   26358.9711  force_treeallocate()/forcetree.c/4811
  11 2979 0          Nextnode    222.3021   26581.2732  force_treeallocate()/forcetree.c/4820
 126 2982 0         Dist2list    126.8599   26994.6948  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 6
   3 2981 0           Ngblist     63.3994   26866.6202  subfind_density()/subfind_density.c/78
   3 2982 0         Dist2list    126.7988   26993.4190  subfind_density()/subfind_density.c/79
   3 2983 0    DataIndexTable     21.8182   27015.2372  subfind_density()/subfind_density.c/87
 118 2981 0           Ngblist     63.4314   26868.8376  subfind_density()/subfind_density.c/78
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 122
  14 2990 0             DM_Vz    126.8516   27725.8147  subfind_density()/subfind_density.c/97
  14 2991 0    	  DensDataGet    137.2408   27863.0555  subfind_density()/subfind_density.c/161
  14 2992 0  	  DensDataResult    102.9306   27965.9862  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   4 2974 0         FOF_GList    634.0635   19495.3153  fof_fof()/fof.c/337
   4 2975 0             Group     12.5103   19507.8257  fof_fof()/fof.c/380
   4 2976 0   DomainNodeIndex      0.3469   19508.1726  force_treeallocate()/forcetree.c/4800
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 26
  11 2980 0            Father    221.9056   26803.1788  force_treeallocate()/forcetree.c/4827
  11 2981 0           Ngblist     63.4288   26866.6076  subfind_density()/subfind_density.c/78
  11 2982 0         Dist2list    126.8576   26993.4653  subfind_density()/subfind_density.c/79
  11 2983 0    DataIndexTable     21.8182   27015.2834  subfind_density()/subfind_density.c/87
 126 2983 0    DataIndexTable     21.8182   27016.5130  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   3 2984 0      DataNodeList     58.1818   27073.4190  subfind_density()/subfind_density.c/89
   3 2985 0              Left    126.7988   27200.2178  subfind_density()/subfind_density.c/91
   3 2986 0             Right    126.7988   27327.0166  subfind_density()/subfind_density.c/92
   3 2987 0              Todo     15.8499   27342.8665  subfind_density()/subfind_density.c/93
 118 2982 0         Dist2list    126.8628   26995.7004  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=14: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=34.0138 MB).
   4 2977 0        Nodes_base   3737.5296   23245.7022  force_treeallocate()/forcetree.c/4803
   4 2978 0     Extnodes_base   3114.6080   26360.3102  force_treeallocate()/forcetree.c/4811
   4 2979 0          Nextnode    222.3021   26582.6123  force_treeallocate()/forcetree.c/4820
  11 2984 0      DataNodeList     58.1818   27073.4652  subfind_density()/subfind_density.c/89
  11 2985 0              Left    126.8576   27200.3229  subfind_density()/subfind_density.c/91
  11 2986 0             Right    126.8576   27327.1805  subfind_density()/subfind_density.c/92
  11 2987 0              Todo     15.8572   27343.0377  subfind_density()/subfind_density.c/93
 126 2984 0      DataNodeList     58.1818   27074.6948  subfind_density()/subfind_density.c/89
   7 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   7 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   7 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   7 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   7 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   7 2973 0         FOF_PList    380.5332   18861.3469  fof_fof()/fof.c/249
   7 2974 0         FOF_GList    634.2220   19495.5689  fof_fof()/fof.c/337
   7 2975 0             Group     10.4154   19505.9843  fof_fof()/fof.c/380
   7 2976 0   DomainNodeIndex      0.3469   19506.3313  force_treeallocate()/forcetree.c/4800
   7 2977 0        Nodes_base   3737.5296   23243.8609  force_treeallocate()/forcetree.c/4803
   7 2978 0     Extnodes_base   3114.6080   26358.4689  force_treeallocate()/forcetree.c/4811
   7 2979 0          Nextnode    222.3021   26580.7710  force_treeallocate()/forcetree.c/4820
   7 2980 0            Father    221.9056   26802.6765  force_treeallocate()/forcetree.c/4827
   7 2981 0           Ngblist     63.4222   26866.0987  subfind_density()/subfind_density.c/78
   7 2982 0         Dist2list    126.8444   26992.9432  subfind_density()/subfind_density.c/79
   7 2983 0    DataIndexTable     21.8182   27014.7613  subfind_density()/subfind_density.c/87
   7 2984 0      DataNodeList     58.1818   27072.9431  subfind_density()/subfind_density.c/89
   7 2985 0              Left    126.8444   27199.7875  subfind_density()/subfind_density.c/91
   3 2988 0             DM_Vx    126.7988   27469.6653  subfind_density()/subfind_density.c/95
   3 2989 0             DM_Vy    126.7988   27596.4641  subfind_density()/subfind_density.c/96
   3 2990 0             DM_Vz    126.7988   27723.2630  subfind_density()/subfind_density.c/97
 118 2983 0    DataIndexTable     21.8182   27017.5186  subfind_density()/subfind_density.c/87
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
   1 2973 0         FOF_PList    380.5427   18861.3564  fof_fof()/fof.c/249
   1 2974 0         FOF_GList    634.2378   19495.5942  fof_fof()/fof.c/337
   1 2975 0             Group     11.8483   19507.4425  fof_fof()/fof.c/380
   1 2976 0   DomainNodeIndex      0.3469   19507.7895  force_treeallocate()/forcetree.c/4800
   1 2977 0        Nodes_base   3737.5296   23245.3191  force_treeallocate()/forcetree.c/4803
   1 2978 0     Extnodes_base   3114.6080   26359.9271  force_treeallocate()/forcetree.c/4811
 123 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 123 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 123 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 123 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 123 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 123 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 123 2973 0         FOF_PList    380.5779   18861.3915  fof_fof()/fof.c/249
 123 2974 0         FOF_GList    634.2964   19495.6880  fof_fof()/fof.c/337
 123 2975 0             Group     13.2346   19508.9225  fof_fof()/fof.c/380
 123 2976 0   DomainNodeIndex      0.3469   19509.2694  force_treeallocate()/forcetree.c/4800
 123 2977 0        Nodes_base   3737.5296   23246.7990  force_treeallocate()/forcetree.c/4803
task 14: endrun called with an error level of 812


   4 2980 0            Father    221.9056   26804.5179  force_treeallocate()/forcetree.c/4827
   4 2981 0           Ngblist     63.4064   26867.9242  subfind_density()/subfind_density.c/78
  11 2988 0             DM_Vx    126.8576   27469.8954  subfind_density()/subfind_density.c/95
  11 2989 0             DM_Vy    126.8576   27596.7530  subfind_density()/subfind_density.c/96
  11 2990 0             DM_Vz    126.8576   27723.6107  subfind_density()/subfind_density.c/97
 126 2985 0              Left    126.8599   27201.5548  subfind_density()/subfind_density.c/91
   7 2986 0             Right    126.8444   27326.6319  subfind_density()/subfind_density.c/92
   7 2987 0              Todo     15.8556   27342.4875  subfind_density()/subfind_density.c/93
   7 2988 0             DM_Vx    126.8444   27469.3319  subfind_density()/subfind_density.c/95
   7 2989 0             DM_Vy    126.8444   27596.1763  subfind_density()/subfind_density.c/96
   7 2990 0             DM_Vz    126.8444   27723.0207  subfind_density()/subfind_density.c/97
   7 2991 0    	  DensDataGet    116.1394   27839.1601  subfind_density()/subfind_density.c/161
   3 2991 0    	  DensDataGet    144.3306   27867.5936  subfind_density()/subfind_density.c/161
   3 2992 0  	  DensDataResult    108.2480   27975.8416  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=3: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=24.1584 MB).
task 3: endrun called with an error level of 812


 118 2984 0      DataNodeList     58.1818   27075.7004  subfind_density()/subfind_density.c/89
   1 2979 0          Nextnode    222.3021   26582.2291  force_treeallocate()/forcetree.c/4820
   1 2980 0            Father    221.9056   26804.1347  force_treeallocate()/forcetree.c/4827
 123 2978 0     Extnodes_base   3114.6080   26361.4071  force_treeallocate()/forcetree.c/4811
 123 2979 0          Nextnode    222.3021   26583.7091  force_treeallocate()/forcetree.c/4820
 123 2980 0            Father    221.9056   26805.6147  force_treeallocate()/forcetree.c/4827
 123 2981 0           Ngblist     63.4296   26869.0444  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 14
   4 2982 0         Dist2list    126.8127   26994.7369  subfind_density()/subfind_density.c/79
  11 2991 0    	  DensDataGet    128.4019   27852.0126  subfind_density()/subfind_density.c/161
  11 2992 0  	  DensDataResult     96.3014   27948.3140  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 126 2986 0             Right    126.8599   27328.4147  subfind_density()/subfind_density.c/92
   7 2992 0  	  DensDataResult     87.1046   27926.2647  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=7: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.7353 MB).
task 7: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 3
 118 2985 0              Left    126.8628   27202.5632  subfind_density()/subfind_density.c/91
   1 2981 0           Ngblist     63.4238   26867.5585  subfind_density()/subfind_density.c/78
 123 2982 0         Dist2list    126.8593   26995.9036  subfind_density()/subfind_density.c/79
 123 2983 0    DataIndexTable     21.8182   27017.7218  subfind_density()/subfind_density.c/87
 123 2984 0      DataNodeList     58.1818   27075.9036  subfind_density()/subfind_density.c/89
   4 2983 0    DataIndexTable     21.8182   27016.5551  subfind_density()/subfind_density.c/87

Task=11: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.686 MB).
task 11: endrun called with an error level of 812


 126 2987 0              Todo     15.8575   27344.2722  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 7
 118 2986 0             Right    126.8628   27329.4260  subfind_density()/subfind_density.c/92
   1 2982 0         Dist2list    126.8476   26994.4061  subfind_density()/subfind_density.c/79
 123 2985 0              Left    126.8593   27202.7629  subfind_density()/subfind_density.c/91
 123 2986 0             Right    126.8593   27329.6222  subfind_density()/subfind_density.c/92
 123 2987 0              Todo     15.8574   27345.4796  subfind_density()/subfind_density.c/93
   4 2984 0      DataNodeList     58.1818   27074.7369  subfind_density()/subfind_density.c/89
   4 2985 0              Left    126.8127   27201.5496  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 11
 126 2988 0             DM_Vx    126.8599   27471.1321  subfind_density()/subfind_density.c/95
 118 2987 0              Todo     15.8578   27345.2838  subfind_density()/subfind_density.c/93
   1 2983 0    DataIndexTable     21.8182   27016.2243  subfind_density()/subfind_density.c/87
 123 2988 0             DM_Vx    126.8593   27472.3389  subfind_density()/subfind_density.c/95
 123 2989 0             DM_Vy    126.8593   27599.1982  subfind_density()/subfind_density.c/96
 123 2990 0             DM_Vz    126.8593   27726.0574  subfind_density()/subfind_density.c/97
   4 2986 0             Right    126.8127   27328.3623  subfind_density()/subfind_density.c/92
   4 2987 0              Todo     15.8516   27344.2139  subfind_density()/subfind_density.c/93
 126 2989 0             DM_Vy    126.8599   27597.9921  subfind_density()/subfind_density.c/96
 118 2988 0             DM_Vx    126.8628   27472.1466  subfind_density()/subfind_density.c/95
   1 2984 0      DataNodeList     58.1818   27074.4061  subfind_density()/subfind_density.c/89
 123 2991 0    	  DensDataGet    117.2606   27843.3180  subfind_density()/subfind_density.c/161
   4 2988 0             DM_Vx    126.8127   27471.0266  subfind_density()/subfind_density.c/95
 126 2990 0             DM_Vz    126.8599   27724.8520  subfind_density()/subfind_density.c/97
 118 2989 0             DM_Vy    126.8628   27599.0094  subfind_density()/subfind_density.c/96
   1 2985 0              Left    126.8476   27201.2536  subfind_density()/subfind_density.c/91
   1 2986 0             Right    126.8476   27328.1012  subfind_density()/subfind_density.c/92
 123 2992 0  	  DensDataResult     87.9454   27931.2634  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   4 2989 0             DM_Vy    126.8127   27597.8393  subfind_density()/subfind_density.c/96
 126 2991 0    	  DensDataGet    115.9882   27840.8402  subfind_density()/subfind_density.c/161
 118 2990 0             DM_Vz    126.8628   27725.8722  subfind_density()/subfind_density.c/97
   1 2987 0              Todo     15.8559   27343.9571  subfind_density()/subfind_density.c/93

Task=123: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.7366 MB).
   4 2990 0             DM_Vz    126.8127   27724.6520  subfind_density()/subfind_density.c/97
 126 2992 0  	  DensDataResult     86.9912   27927.8314  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 118 2991 0    	  DensDataGet    129.0149   27854.8871  subfind_density()/subfind_density.c/161
   1 2988 0             DM_Vx    126.8476   27470.8047  subfind_density()/subfind_density.c/95
task 123: endrun called with an error level of 812


   4 2991 0    	  DensDataGet    118.1859   27842.8379  subfind_density()/subfind_density.c/161
   4 2992 0  	  DensDataResult     88.6394   27931.4772  subfind_density()/subfind_density.c/201

Task=126: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.1686 MB).
task 126: endrun called with an error level of 812


 118 2992 0  	  DensDataResult     96.7612   27951.6483  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   1 2989 0             DM_Vy    126.8476   27597.6523  subfind_density()/subfind_density.c/96
   1 2990 0             DM_Vz    126.8476   27724.4998  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 123
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 126

Task=118: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=48.3517 MB).
task 118: endrun called with an error level of 812


   1 2991 0    	  DensDataGet    109.1097   27833.6095  subfind_density()/subfind_density.c/161

Task=4: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.5228 MB).
task 4: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 118
   1 2992 0  	  DensDataResult     81.8323   27915.4418  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 4

Task=1: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.5582 MB).
task 1: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 1
 119 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 119 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 119 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 119 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 119 2973 0         FOF_PList    380.5831   18861.3968  fof_fof()/fof.c/249
 119 2974 0         FOF_GList    634.3052   19495.7020  fof_fof()/fof.c/337
 119 2975 0             Group     11.0712   19506.7732  fof_fof()/fof.c/380
 119 2976 0   DomainNodeIndex      0.3469   19507.1201  force_treeallocate()/forcetree.c/4800
 119 2977 0        Nodes_base   3737.5296   23244.6497  force_treeallocate()/forcetree.c/4803
 119 2978 0     Extnodes_base   3114.6080   26359.2577  force_treeallocate()/forcetree.c/4811
 119 2979 0          Nextnode    222.3021   26581.5598  force_treeallocate()/forcetree.c/4820
 119 2980 0            Father    221.9056   26803.4654  force_treeallocate()/forcetree.c/4827
 119 2981 0           Ngblist     63.4305   26866.8959  subfind_density()/subfind_density.c/78
 119 2982 0         Dist2list    126.8610   26993.7569  subfind_density()/subfind_density.c/79
 119 2983 0    DataIndexTable     21.8182   27015.5751  subfind_density()/subfind_density.c/87
 119 2984 0      DataNodeList     58.1818   27073.7569  subfind_density()/subfind_density.c/89
 119 2985 0              Left    126.8610   27200.6179  subfind_density()/subfind_density.c/91
 119 2986 0             Right    126.8610   27327.4790  subfind_density()/subfind_density.c/92
 119 2987 0              Todo     15.8576   27343.3366  subfind_density()/subfind_density.c/93
 119 2988 0             DM_Vx    126.8610   27470.1976  subfind_density()/subfind_density.c/95
 119 2989 0             DM_Vy    126.8610   27597.0587  subfind_density()/subfind_density.c/96
 119 2990 0             DM_Vz    126.8610   27723.9197  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 119 2991 0    	  DensDataGet    140.1671   27864.0868  subfind_density()/subfind_density.c/161
  13 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  13 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  13 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  13 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  13 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  13 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  13 2973 0         FOF_PList    380.5380   18861.3517  fof_fof()/fof.c/249
  13 2974 0         FOF_GList    634.2301   19495.5818  fof_fof()/fof.c/337
  13 2975 0             Group     12.4294   19508.0112  fof_fof()/fof.c/380
  13 2976 0   DomainNodeIndex      0.3469   19508.3581  force_treeallocate()/forcetree.c/4800
  13 2977 0        Nodes_base   3737.5296   23245.8877  force_treeallocate()/forcetree.c/4803
  13 2978 0     Extnodes_base   3114.6080   26360.4957  force_treeallocate()/forcetree.c/4811
  13 2979 0          Nextnode    222.3021   26582.7978  force_treeallocate()/forcetree.c/4820
  13 2980 0            Father    221.9056   26804.7034  force_treeallocate()/forcetree.c/4827
  13 2981 0           Ngblist     63.4230   26868.1264  subfind_density()/subfind_density.c/78
  13 2982 0         Dist2list    126.8460   26994.9724  subfind_density()/subfind_density.c/79
 114 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 114 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 114 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 114 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 114 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 114 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 114 2973 0         FOF_PList    380.2452   18861.0589  fof_fof()/fof.c/249
 114 2974 0         FOF_GList    633.7421   19494.8010  fof_fof()/fof.c/337
 114 2975 0             Group     12.5093   19507.3103  fof_fof()/fof.c/380
 114 2976 0   DomainNodeIndex      0.3469   19507.6572  force_treeallocate()/forcetree.c/4800
 114 2977 0        Nodes_base   3737.5296   23245.1868  force_treeallocate()/forcetree.c/4803
 119 2992 0  	  DensDataResult    105.1253   27969.2121  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  13 2983 0    DataIndexTable     21.8182   27016.7906  subfind_density()/subfind_density.c/87
  13 2984 0      DataNodeList     58.1818   27074.9724  subfind_density()/subfind_density.c/89
  13 2985 0              Left    126.8460   27201.8184  subfind_density()/subfind_density.c/91
  13 2986 0             Right    126.8460   27328.6644  subfind_density()/subfind_density.c/92
 114 2978 0     Extnodes_base   3114.6080   26359.7948  force_treeallocate()/forcetree.c/4811

Task=119: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=30.7879 MB).
task 119: endrun called with an error level of 812


 100 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 100 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 100 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 100 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 100 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  27 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  27 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  27 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  27 2973 0         FOF_PList    379.2083   18860.0220  fof_fof()/fof.c/249
  27 2974 0         FOF_GList    632.0139   19492.0359  fof_fof()/fof.c/337
  27 2975 0             Group     10.4154   19502.4513  fof_fof()/fof.c/380
  27 2976 0   DomainNodeIndex      0.3469   19502.7982  force_treeallocate()/forcetree.c/4800
  27 2977 0        Nodes_base   3737.5296   23240.3278  force_treeallocate()/forcetree.c/4803
  27 2978 0     Extnodes_base   3114.6080   26354.9358  force_treeallocate()/forcetree.c/4811
  27 2979 0          Nextnode    222.3021   26577.2379  force_treeallocate()/forcetree.c/4820
  13 2987 0              Todo     15.8558   27344.5202  subfind_density()/subfind_density.c/93
  13 2988 0             DM_Vx    126.8460   27471.3662  subfind_density()/subfind_density.c/95
  13 2989 0             DM_Vy    126.8460   27598.2122  subfind_density()/subfind_density.c/96
  13 2990 0             DM_Vz    126.8460   27725.0582  subfind_density()/subfind_density.c/97
 114 2979 0          Nextnode    222.3021   26582.0969  force_treeallocate()/forcetree.c/4820
 114 2980 0            Father    221.9056   26804.0025  force_treeallocate()/forcetree.c/4827
 114 2981 0           Ngblist     63.3742   26867.3767  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 119
 100 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 100 2973 0         FOF_PList    380.4616   18861.2753  fof_fof()/fof.c/249
 100 2974 0         FOF_GList    634.1027   19495.3780  fof_fof()/fof.c/337
 100 2975 0             Group     11.2963   19506.6743  fof_fof()/fof.c/380
  27 2980 0            Father    221.9056   26799.1435  force_treeallocate()/forcetree.c/4827
  27 2981 0           Ngblist     63.2014   26862.3448  subfind_density()/subfind_density.c/78
  27 2982 0         Dist2list    126.4028   26988.7476  subfind_density()/subfind_density.c/79
  27 2983 0    DataIndexTable     21.8182   27010.5658  subfind_density()/subfind_density.c/87
  13 2991 0    	  DensDataGet    117.7266   27842.7849  subfind_density()/subfind_density.c/161
  13 2992 0  	  DensDataResult     88.2950   27931.0798  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 114 2982 0         Dist2list    126.7484   26994.1251  subfind_density()/subfind_density.c/79
 114 2983 0    DataIndexTable     21.8182   27015.9433  subfind_density()/subfind_density.c/87
 114 2984 0      DataNodeList     58.1818   27074.1251  subfind_density()/subfind_density.c/89
 100 2976 0   DomainNodeIndex      0.3469   19507.0212  force_treeallocate()/forcetree.c/4800
 100 2977 0        Nodes_base   3737.5296   23244.5508  force_treeallocate()/forcetree.c/4803
 100 2978 0     Extnodes_base   3114.6080   26359.1588  force_treeallocate()/forcetree.c/4811
 100 2979 0          Nextnode    222.3021   26581.4609  force_treeallocate()/forcetree.c/4820
  27 2984 0      DataNodeList     58.1818   27068.7476  subfind_density()/subfind_density.c/89
  27 2985 0              Left    126.4028   27195.1504  subfind_density()/subfind_density.c/91
  27 2986 0             Right    126.4028   27321.5531  subfind_density()/subfind_density.c/92
  27 2987 0              Todo     15.8003   27337.3535  subfind_density()/subfind_density.c/93

Task=13: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.9202 MB).
task 13: endrun called with an error level of 812


 114 2985 0              Left    126.7484   27200.8735  subfind_density()/subfind_density.c/91
 114 2986 0             Right    126.7484   27327.6219  subfind_density()/subfind_density.c/92
 100 2980 0            Father    221.9056   26803.3665  force_treeallocate()/forcetree.c/4827
 100 2981 0           Ngblist     63.4103   26866.7768  subfind_density()/subfind_density.c/78
  27 2988 0             DM_Vx    126.4028   27463.7562  subfind_density()/subfind_density.c/95
  27 2989 0             DM_Vy    126.4028   27590.1590  subfind_density()/subfind_density.c/96
  27 2990 0             DM_Vz    126.4028   27716.5618  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 13
 114 2987 0              Todo     15.8436   27343.4654  subfind_density()/subfind_density.c/93
 100 2982 0         Dist2list    126.8205   26993.5973  subfind_density()/subfind_density.c/79
 100 2983 0    DataIndexTable     21.8182   27015.4155  subfind_density()/subfind_density.c/87
 100 2984 0      DataNodeList     58.1818   27073.5973  subfind_density()/subfind_density.c/89
  27 2991 0    	  DensDataGet    128.5878   27845.1496  subfind_density()/subfind_density.c/161
  27 2992 0  	  DensDataResult     96.4409   27941.5905  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 114 2988 0             DM_Vx    126.7484   27470.2139  subfind_density()/subfind_density.c/95
 100 2985 0              Left    126.8205   27200.4178  subfind_density()/subfind_density.c/91
 100 2986 0             Right    126.8205   27327.2383  subfind_density()/subfind_density.c/92
 100 2987 0              Todo     15.8526   27343.0909  subfind_density()/subfind_density.c/93

Task=27: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.4095 MB).
task 27: endrun called with an error level of 812


 114 2989 0             DM_Vy    126.7484   27596.9623  subfind_density()/subfind_density.c/96
 100 2988 0             DM_Vx    126.8205   27469.9114  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 27
 114 2990 0             DM_Vz    126.7484   27723.7107  subfind_density()/subfind_density.c/97
 100 2989 0             DM_Vy    126.8205   27596.7320  subfind_density()/subfind_density.c/96
 114 2991 0    	  DensDataGet    135.9307   27859.6414  subfind_density()/subfind_density.c/161
 100 2990 0             DM_Vz    126.8205   27723.5525  subfind_density()/subfind_density.c/97
 114 2992 0  	  DensDataResult    101.9480   27961.5894  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------- 100 2991 0    	  DensDataGet    134.6741   27858.2266  subfind_density()/subfind_density.c/161

 100 2992 0  	  DensDataResult    101.0056   27959.2322  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=114: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=38.4106 MB).

Task=100: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=40.7678 MB).
task 100: endrun called with an error level of 812


task 114: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 100
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 114
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  32 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  32 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  32 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  32 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  32 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  32 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  32 2973 0         FOF_PList    380.5171   18861.3308  fof_fof()/fof.c/249
  32 2974 0         FOF_GList    634.1952   19495.5260  fof_fof()/fof.c/337
  32 2975 0             Group     11.2320   19506.7580  fof_fof()/fof.c/380
  32 2976 0   DomainNodeIndex      0.3469   19507.1049  force_treeallocate()/forcetree.c/4800
  32 2977 0        Nodes_base   3737.5296   23244.6345  force_treeallocate()/forcetree.c/4803
  32 2978 0     Extnodes_base   3114.6080   26359.2425  force_treeallocate()/forcetree.c/4811
  32 2979 0          Nextnode    222.3021   26581.5446  force_treeallocate()/forcetree.c/4820
  32 2980 0            Father    221.9056   26803.4502  force_treeallocate()/forcetree.c/4827
  32 2981 0           Ngblist     63.4195   26866.8697  subfind_density()/subfind_density.c/78
  32 2982 0         Dist2list    126.8390   26993.7087  subfind_density()/subfind_density.c/79
  32 2983 0    DataIndexTable     21.8182   27015.5269  subfind_density()/subfind_density.c/87
  32 2984 0      DataNodeList     58.1818   27073.7087  subfind_density()/subfind_density.c/89
  32 2985 0              Left    126.8390   27200.5477  subfind_density()/subfind_density.c/91
  32 2986 0             Right    126.8390   27327.3868  subfind_density()/subfind_density.c/92
  32 2987 0              Todo     15.8549   27343.2416  subfind_density()/subfind_density.c/93
  32 2988 0             DM_Vx    126.8390   27470.0807  subfind_density()/subfind_density.c/95
  32 2989 0             DM_Vy    126.8390   27596.9197  subfind_density()/subfind_density.c/96
  32 2990 0             DM_Vz    126.8390   27723.7587  subfind_density()/subfind_density.c/97
  32 2991 0    	  DensDataGet    116.7803   27840.5391  subfind_density()/subfind_density.c/161
  32 2992 0  	  DensDataResult     87.5853   27928.1243  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=32: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.8757 MB).
task 32: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 32
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 32:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 32: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 32
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  44 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  44 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  44 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  44 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  44 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  44 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  44 2973 0         FOF_PList    380.5363   18861.3500  fof_fof()/fof.c/249
  44 2974 0         FOF_GList    634.2272   19495.5772  fof_fof()/fof.c/337
  44 2975 0             Group     12.1451   19507.7223  fof_fof()/fof.c/380
  44 2976 0   DomainNodeIndex      0.3469   19508.0693  force_treeallocate()/forcetree.c/4800
  44 2977 0        Nodes_base   3737.5296   23245.5989  force_treeallocate()/forcetree.c/4803
  44 2978 0     Extnodes_base   3114.6080   26360.2069  force_treeallocate()/forcetree.c/4811
  44 2979 0          Nextnode    222.3021   26582.5089  force_treeallocate()/forcetree.c/4820
  44 2980 0            Father    221.9056   26804.4145  force_treeallocate()/forcetree.c/4827
  44 2981 0           Ngblist     63.4227   26867.8372  subfind_density()/subfind_density.c/78
  44 2982 0         Dist2list    126.8454   26994.6827  subfind_density()/subfind_density.c/79
  44 2983 0    DataIndexTable     21.8182   27016.5009  subfind_density()/subfind_density.c/87
  44 2984 0      DataNodeList     58.1818   27074.6827  subfind_density()/subfind_density.c/89
  44 2985 0              Left    126.8454   27201.5281  subfind_density()/subfind_density.c/91
  44 2986 0             Right    126.8454   27328.3735  subfind_density()/subfind_density.c/92
  44 2987 0              Todo     15.8557   27344.2292  subfind_density()/subfind_density.c/93
  44 2988 0             DM_Vx    126.8454   27471.0747  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  44 2989 0             DM_Vy    126.8454   27597.9201  subfind_density()/subfind_density.c/96
  44 2990 0             DM_Vz    126.8454   27724.7656  subfind_density()/subfind_density.c/97
  44 2991 0    	  DensDataGet    111.6450   27836.4106  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  82 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  82 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  82 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  82 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  82 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  82 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  82 2973 0         FOF_PList    380.4627   18861.2764  fof_fof()/fof.c/249
  82 2974 0         FOF_GList    634.1045   19495.3810  fof_fof()/fof.c/337
  82 2975 0             Group     10.9207   19506.3017  fof_fof()/fof.c/380
  82 2976 0   DomainNodeIndex      0.3469   19506.6486  force_treeallocate()/forcetree.c/4800
  82 2977 0        Nodes_base   3737.5296   23244.1782  force_treeallocate()/forcetree.c/4803
  82 2978 0     Extnodes_base   3114.6080   26358.7862  force_treeallocate()/forcetree.c/4811
  82 2979 0          Nextnode    222.3021   26581.0883  force_treeallocate()/forcetree.c/4820
  44 2992 0  	  DensDataResult     83.7338   27920.1443  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 102 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 102 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 102 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 102 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 102 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 102 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 102 2973 0         FOF_PList    380.5499   18861.3636  fof_fof()/fof.c/249
 102 2974 0         FOF_GList    634.2498   19495.6134  fof_fof()/fof.c/337
 102 2975 0             Group     11.0598   19506.6732  fof_fof()/fof.c/380
 102 2976 0   DomainNodeIndex      0.3469   19507.0201  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  82 2980 0            Father    221.9056   26802.9939  force_treeallocate()/forcetree.c/4827
  82 2981 0           Ngblist     63.4105   26866.4043  subfind_density()/subfind_density.c/78
  82 2982 0         Dist2list    126.8209   26993.2252  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=44: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.8557 MB).
  70 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  70 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  70 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  70 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  70 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  70 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  70 2973 0         FOF_PList    380.5624   18861.3761  fof_fof()/fof.c/249
  70 2974 0         FOF_GList    634.2707   19495.6468  fof_fof()/fof.c/337
  70 2975 0             Group     11.1168   19506.7636  fof_fof()/fof.c/380
  70 2976 0   DomainNodeIndex      0.3469   19507.1106  force_treeallocate()/forcetree.c/4800
  70 2977 0        Nodes_base   3737.5296   23244.6402  force_treeallocate()/forcetree.c/4803
  70 2978 0     Extnodes_base   3114.6080   26359.2482  force_treeallocate()/forcetree.c/4811
  64 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  64 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  64 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  64 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  64 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  64 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  64 2973 0         FOF_PList    380.3812   18861.1949  fof_fof()/fof.c/249
  64 2974 0         FOF_GList    633.9687   19495.1637  fof_fof()/fof.c/337
  64 2975 0             Group     13.7482   19508.9118  fof_fof()/fof.c/380
  64 2976 0   DomainNodeIndex      0.3469   19509.2588  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 102 2977 0        Nodes_base   3737.5296   23244.5497  force_treeallocate()/forcetree.c/4803
 102 2978 0     Extnodes_base   3114.6080   26359.1577  force_treeallocate()/forcetree.c/4811
  63 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  63 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  63 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  63 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  63 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  63 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  63 2973 0         FOF_PList    380.4192   18861.2329  fof_fof()/fof.c/249
  63 2974 0         FOF_GList    634.0321   19495.2650  fof_fof()/fof.c/337
  63 2975 0             Group     12.0351   19507.3001  fof_fof()/fof.c/380
  63 2976 0   DomainNodeIndex      0.3469   19507.6470  force_treeallocate()/forcetree.c/4800
  63 2977 0        Nodes_base   3737.5296   23245.1766  force_treeallocate()/forcetree.c/4803
  63 2978 0     Extnodes_base   3114.6080   26359.7846  force_treeallocate()/forcetree.c/4811
  63 2979 0          Nextnode    222.3021   26582.0867  force_treeallocate()/forcetree.c/4820
  63 2980 0            Father    221.9056   26803.9923  force_treeallocate()/forcetree.c/4827
  63 2981 0           Ngblist     63.4032   26867.3955  subfind_density()/subfind_density.c/78
  82 2983 0    DataIndexTable     21.8182   27015.0434  subfind_density()/subfind_density.c/87
  82 2984 0      DataNodeList     58.1818   27073.2252  subfind_density()/subfind_density.c/89
  82 2985 0              Left    126.8209   27200.0461  subfind_density()/subfind_density.c/91
 115 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 115 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 115 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 115 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 115 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 115 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 115 2973 0         FOF_PList    380.5758   18861.3895  fof_fof()/fof.c/249
 115 2974 0         FOF_GList    634.2929   19495.6824  fof_fof()/fof.c/337
 115 2975 0             Group     13.5386   19509.2210  fof_fof()/fof.c/380
 115 2976 0   DomainNodeIndex      0.3469   19509.5679  force_treeallocate()/forcetree.c/4800
 115 2977 0        Nodes_base   3737.5296   23247.0975  force_treeallocate()/forcetree.c/4803
 115 2978 0     Extnodes_base   3114.6080   26361.7055  force_treeallocate()/forcetree.c/4811
 115 2979 0          Nextnode    222.3021   26584.0076  force_treeallocate()/forcetree.c/4820
 115 2980 0            Father    221.9056   26805.9132  force_treeallocate()/forcetree.c/4827
task 44: endrun called with an error level of 812


  70 2979 0          Nextnode    222.3021   26581.5503  force_treeallocate()/forcetree.c/4820
  70 2980 0            Father    221.9056   26803.4558  force_treeallocate()/forcetree.c/4827
  64 2977 0        Nodes_base   3737.5296   23246.7884  force_treeallocate()/forcetree.c/4803
  64 2978 0     Extnodes_base   3114.6080   26361.3964  force_treeallocate()/forcetree.c/4811
  64 2979 0          Nextnode    222.3021   26583.6984  force_treeallocate()/forcetree.c/4820
  57 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  57 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  57 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  57 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  57 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  57 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  57 2973 0         FOF_PList    380.0305   18860.8442  fof_fof()/fof.c/249
  57 2974 0         FOF_GList    633.3841   19494.2283  fof_fof()/fof.c/337
  57 2975 0             Group     11.4758   19505.7041  fof_fof()/fof.c/380
  57 2976 0   DomainNodeIndex      0.3469   19506.0511  force_treeallocate()/forcetree.c/4800
  57 2977 0        Nodes_base   3737.5296   23243.5807  force_treeallocate()/forcetree.c/4803
  57 2978 0     Extnodes_base   3114.6080   26358.1887  force_treeallocate()/forcetree.c/4811
  57 2979 0          Nextnode    222.3021   26580.4907  force_treeallocate()/forcetree.c/4820
  57 2980 0            Father    221.9056   26802.3963  force_treeallocate()/forcetree.c/4827
  57 2981 0           Ngblist     63.3384   26865.7347  subfind_density()/subfind_density.c/78
  57 2982 0         Dist2list    126.6768   26992.4116  subfind_density()/subfind_density.c/79
  57 2983 0    DataIndexTable     21.8182   27014.2298  subfind_density()/subfind_density.c/87
  57 2984 0      DataNodeList     58.1818   27072.4115  subfind_density()/subfind_density.c/89
  57 2985 0              Left    126.6768   27199.0884  subfind_density()/subfind_density.c/91
 102 2979 0          Nextnode    222.3021   26581.4598  force_treeallocate()/forcetree.c/4820
 102 2980 0            Father    221.9056   26803.3654  force_treeallocate()/forcetree.c/4827
 102 2981 0           Ngblist     63.4250   26866.7904  subfind_density()/subfind_density.c/78
  63 2982 0         Dist2list    126.8064   26994.2019  subfind_density()/subfind_density.c/79
  63 2983 0    DataIndexTable     21.8182   27016.0201  subfind_density()/subfind_density.c/87
  63 2984 0      DataNodeList     58.1818   27074.2019  subfind_density()/subfind_density.c/89
  63 2985 0              Left    126.8064   27201.0083  subfind_density()/subfind_density.c/91
  63 2986 0             Right    126.8064   27327.8147  subfind_density()/subfind_density.c/92
  82 2986 0             Right    126.8209   27326.8670  subfind_density()/subfind_density.c/92
  82 2987 0              Todo     15.8526   27342.7196  subfind_density()/subfind_density.c/93
  82 2988 0             DM_Vx    126.8209   27469.5405  subfind_density()/subfind_density.c/95
 115 2981 0           Ngblist     63.4293   26869.3425  subfind_density()/subfind_density.c/78
 115 2982 0         Dist2list    126.8586   26996.2011  subfind_density()/subfind_density.c/79
 115 2983 0    DataIndexTable     21.8182   27018.0192  subfind_density()/subfind_density.c/87
 115 2984 0      DataNodeList     58.1818   27076.2010  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 44
  70 2981 0           Ngblist     63.4271   26866.8829  subfind_density()/subfind_density.c/78
  70 2982 0         Dist2list    126.8541   26993.7371  subfind_density()/subfind_density.c/79
  70 2983 0    DataIndexTable     21.8182   27015.5552  subfind_density()/subfind_density.c/87
  64 2980 0            Father    221.9056   26805.6040  force_treeallocate()/forcetree.c/4827
  64 2981 0           Ngblist     63.3969   26869.0009  subfind_density()/subfind_density.c/78
  64 2982 0         Dist2list    126.7937   26995.7947  subfind_density()/subfind_density.c/79
  57 2986 0             Right    126.6768   27325.7652  subfind_density()/subfind_density.c/92
  57 2987 0              Todo     15.8346   27341.5998  subfind_density()/subfind_density.c/93
  57 2988 0             DM_Vx    126.6768   27468.2766  subfind_density()/subfind_density.c/95
  57 2989 0             DM_Vy    126.6768   27594.9535  subfind_density()/subfind_density.c/96
  57 2990 0             DM_Vz    126.6768   27721.6303  subfind_density()/subfind_density.c/97
  57 2991 0    	  DensDataGet    135.5875   27857.2178  subfind_density()/subfind_density.c/161
  57 2992 0  	  DensDataResult    101.6906   27958.9085  subfind_density()/subfind_density.c/201
 102 2982 0         Dist2list    126.8500   26993.6403  subfind_density()/subfind_density.c/79
 102 2983 0    DataIndexTable     21.8182   27015.4585  subfind_density()/subfind_density.c/87
  63 2987 0              Todo     15.8508   27343.6655  subfind_density()/subfind_density.c/93
  63 2988 0             DM_Vx    126.8064   27470.4719  subfind_density()/subfind_density.c/95
  63 2989 0             DM_Vy    126.8064   27597.2783  subfind_density()/subfind_density.c/96
  63 2990 0             DM_Vz    126.8064   27724.0847  subfind_density()/subfind_density.c/97
  82 2989 0             DM_Vy    126.8209   27596.3614  subfind_density()/subfind_density.c/96
  82 2990 0             DM_Vz    126.8209   27723.1824  subfind_density()/subfind_density.c/97
  82 2991 0    	  DensDataGet    120.0667   27843.2491  subfind_density()/subfind_density.c/161
 115 2985 0              Left    126.8586   27203.0596  subfind_density()/subfind_density.c/91
 115 2986 0             Right    126.8586   27329.9182  subfind_density()/subfind_density.c/92
 115 2987 0              Todo     15.8573   27345.7755  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  70 2984 0      DataNodeList     58.1818   27073.7370  subfind_density()/subfind_density.c/89
  70 2985 0              Left    126.8541   27200.5912  subfind_density()/subfind_density.c/91
  70 2986 0             Right    126.8541   27327.4453  subfind_density()/subfind_density.c/92
  64 2983 0    DataIndexTable     21.8182   27017.6128  subfind_density()/subfind_density.c/87
  64 2984 0      DataNodeList     58.1818   27075.7946  subfind_density()/subfind_density.c/89
  64 2985 0              Left    126.7937   27202.5884  subfind_density()/subfind_density.c/91
----------------------------------------------------------------------------------------

Task=57: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=41.0915 MB).
task 57: endrun called with an error level of 812


 102 2984 0      DataNodeList     58.1818   27073.6403  subfind_density()/subfind_density.c/89
 102 2985 0              Left    126.8500   27200.4903  subfind_density()/subfind_density.c/91
 102 2986 0             Right    126.8500   27327.3402  subfind_density()/subfind_density.c/92
  63 2991 0    	  DensDataGet    115.7322   27839.8170  subfind_density()/subfind_density.c/161
  63 2992 0  	  DensDataResult     86.7992   27926.6161  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  82 2992 0  	  DensDataResult     90.0500   27933.2991  subfind_density()/subfind_density.c/201
 115 2988 0             DM_Vx    126.8586   27472.6341  subfind_density()/subfind_density.c/95
 115 2989 0             DM_Vy    126.8586   27599.4927  subfind_density()/subfind_density.c/96
 115 2990 0             DM_Vz    126.8586   27726.3513  subfind_density()/subfind_density.c/97
 115 2991 0    	  DensDataGet    113.2864   27839.6377  subfind_density()/subfind_density.c/161
  45 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  45 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  45 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  45 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  45 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  45 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  45 2973 0         FOF_PList    379.8772   18860.6909  fof_fof()/fof.c/249
  45 2974 0         FOF_GList    633.1287   19493.8195  fof_fof()/fof.c/337
  45 2975 0             Group     11.8743   19505.6938  fof_fof()/fof.c/380
  45 2976 0   DomainNodeIndex      0.3469   19506.0407  force_treeallocate()/forcetree.c/4800
  45 2977 0        Nodes_base   3737.5296   23243.5704  force_treeallocate()/forcetree.c/4803
  45 2978 0     Extnodes_base   3114.6080   26358.1784  force_treeallocate()/forcetree.c/4811
  45 2979 0          Nextnode    222.3021   26580.4804  force_treeallocate()/forcetree.c/4820
  98 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  98 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  98 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  98 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  98 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  98 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  98 2973 0         FOF_PList    380.0986   18860.9123  fof_fof()/fof.c/249
  98 2974 0         FOF_GList    633.4977   19494.4101  fof_fof()/fof.c/337
  70 2987 0              Todo     15.8568   27343.3021  subfind_density()/subfind_density.c/93
  70 2988 0             DM_Vx    126.8541   27470.1562  subfind_density()/subfind_density.c/95
  64 2986 0             Right    126.7937   27329.3821  subfind_density()/subfind_density.c/92
  64 2987 0              Todo     15.8492   27345.2313  subfind_density()/subfind_density.c/93
  64 2988 0             DM_Vx    126.7937   27472.0251  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 57
 102 2987 0              Todo     15.8562   27343.1965  subfind_density()/subfind_density.c/93
 102 2988 0             DM_Vx    126.8500   27470.0465  subfind_density()/subfind_density.c/95

Task=63: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.3839 MB).
task 63: endrun called with an error level of 812


----------------------------------------------------------------------------------------
 115 2992 0  	  DensDataResult     84.9648   27924.6025  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=115: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.3975 MB).
task 115: endrun called with an error level of 812


  45 2980 0            Father    221.9056   26802.3860  force_treeallocate()/forcetree.c/4827
  45 2981 0           Ngblist     63.3129   26865.6989  subfind_density()/subfind_density.c/78
  45 2982 0         Dist2list    126.6257   26992.3246  subfind_density()/subfind_density.c/79
  45 2983 0    DataIndexTable     21.8182   27014.1428  subfind_density()/subfind_density.c/87
  45 2984 0      DataNodeList     58.1818   27072.3246  subfind_density()/subfind_density.c/89
  98 2975 0             Group     12.9088   19507.3188  fof_fof()/fof.c/380
  70 2989 0             DM_Vy    126.8541   27597.0104  subfind_density()/subfind_density.c/96
  70 2990 0             DM_Vz    126.8541   27723.8645  subfind_density()/subfind_density.c/97
  64 2989 0             DM_Vy    126.7937   27598.8188  subfind_density()/subfind_density.c/96
 102 2989 0             DM_Vy    126.8500   27596.8964  subfind_density()/subfind_density.c/96
 102 2990 0             DM_Vz    126.8500   27723.7464  subfind_density()/subfind_density.c/97
 102 2991 0    	  DensDataGet    111.3508   27835.0972  subfind_density()/subfind_density.c/161
 102 2992 0  	  DensDataResult     83.5131   27918.6104  subfind_density()/subfind_density.c/201
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 63

Task=82: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.7009 MB).
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 115
  45 2985 0              Left    126.6257   27198.9503  subfind_density()/subfind_density.c/91
  45 2986 0             Right    126.6257   27325.5760  subfind_density()/subfind_density.c/92
  45 2987 0              Todo     15.8282   27341.4043  subfind_density()/subfind_density.c/93
  98 2976 0   DomainNodeIndex      0.3469   19507.6657  force_treeallocate()/forcetree.c/4800
  98 2977 0        Nodes_base   3737.5296   23245.1954  force_treeallocate()/forcetree.c/4803
  98 2978 0     Extnodes_base   3114.6080   26359.8034  force_treeallocate()/forcetree.c/4811
  98 2979 0          Nextnode    222.3021   26582.1054  force_treeallocate()/forcetree.c/4820
  98 2980 0            Father    221.9056   26804.0110  force_treeallocate()/forcetree.c/4827
  70 2991 0    	  DensDataGet    116.1577   27840.0222  subfind_density()/subfind_density.c/161
  70 2992 0  	  DensDataResult     87.1183   27927.1405  subfind_density()/subfind_density.c/201
  64 2990 0             DM_Vz    126.7937   27725.6126  subfind_density()/subfind_density.c/97
----------------------------------------------------------------------------------------
task 82: endrun called with an error level of 812


  45 2988 0             DM_Vx    126.6257   27468.0300  subfind_density()/subfind_density.c/95
  45 2989 0             DM_Vy    126.6257   27594.6557  subfind_density()/subfind_density.c/96
  45 2990 0             DM_Vz    126.6257   27721.2815  subfind_density()/subfind_density.c/97
  98 2981 0           Ngblist     63.3498   26867.3608  subfind_density()/subfind_density.c/78
  98 2982 0         Dist2list    126.6995   26994.0603  subfind_density()/subfind_density.c/79
  98 2983 0    DataIndexTable     21.8182   27015.8785  subfind_density()/subfind_density.c/87
  98 2984 0      DataNodeList     58.1818   27074.0603  subfind_density()/subfind_density.c/89
----------------------------------------------------------------------------------------
  64 2991 0    	  DensDataGet    128.2757   27853.8883  subfind_density()/subfind_density.c/161

Task=102: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.3896 MB).
task 102: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 82
  45 2991 0    	  DensDataGet    129.0994   27850.3809  subfind_density()/subfind_density.c/161
  45 2992 0  	  DensDataResult     96.8246   27947.2055  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  98 2985 0              Left    126.6995   27200.7599  subfind_density()/subfind_density.c/91
  98 2986 0             Right    126.6995   27327.4594  subfind_density()/subfind_density.c/92
  98 2987 0              Todo     15.8374   27343.2969  subfind_density()/subfind_density.c/93

Task=70: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.8595 MB).
task 70: endrun called with an error level of 812


  64 2992 0  	  DensDataResult     96.2068   27950.0950  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 102

Task=45: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.7945 MB).
task 45: endrun called with an error level of 812


  98 2988 0             DM_Vx    126.6995   27469.9964  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 70

application called MPI_Abort(MPI_COMM_WORLD, 812) - process 45
  98 2989 0             DM_Vy    126.6995   27596.6959  subfind_density()/subfind_density.c/96

Task=64: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.905 MB).
  98 2990 0             DM_Vz    126.6995   27723.3955  subfind_density()/subfind_density.c/97
  98 2991 0    	  DensDataGet    131.2975   27854.6930  subfind_density()/subfind_density.c/161
task 64: endrun called with an error level of 812


  98 2992 0  	  DensDataResult     98.4732   27953.1662  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 64

task 64: endrun called with an error level of 124



Task=98: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.8338 MB).
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 64:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 64
task 98: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  65 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 98
  65 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  65 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  65 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  65 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  65 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  65 2973 0         FOF_PList    380.3712   18861.1849  fof_fof()/fof.c/249
  65 2974 0         FOF_GList    633.9520   19495.1368  fof_fof()/fof.c/337
  65 2975 0             Group     10.4154   19505.5522  fof_fof()/fof.c/380
  65 2976 0   DomainNodeIndex      0.3469   19505.8991  force_treeallocate()/forcetree.c/4800
  65 2977 0        Nodes_base   3737.5296   23243.4287  force_treeallocate()/forcetree.c/4803
  65 2978 0     Extnodes_base   3114.6080   26358.0368  force_treeallocate()/forcetree.c/4811
  65 2979 0          Nextnode    222.3021   26580.3388  force_treeallocate()/forcetree.c/4820
  65 2980 0            Father    221.9056   26802.2444  force_treeallocate()/forcetree.c/4827
  65 2981 0           Ngblist     63.3952   26865.6396  subfind_density()/subfind_density.c/78
  65 2982 0         Dist2list    126.7904   26992.4300  subfind_density()/subfind_density.c/79
  65 2983 0    DataIndexTable     21.8182   27014.2482  subfind_density()/subfind_density.c/87
  65 2984 0      DataNodeList     58.1818   27072.4300  subfind_density()/subfind_density.c/89
  65 2985 0              Left    126.7904   27199.2204  subfind_density()/subfind_density.c/91
  65 2986 0             Right    126.7904   27326.0107  subfind_density()/subfind_density.c/92
  65 2987 0              Todo     15.8488   27341.8596  subfind_density()/subfind_density.c/93
  65 2988 0             DM_Vx    126.7904   27468.6499  subfind_density()/subfind_density.c/95
  65 2989 0             DM_Vy    126.7904   27595.4403  subfind_density()/subfind_density.c/96
  65 2990 0             DM_Vz    126.7904   27722.2307  subfind_density()/subfind_density.c/97
  65 2991 0    	  DensDataGet    121.7976   27844.0283  subfind_density()/subfind_density.c/161
  65 2992 0  	  DensDataResult     91.3482   27935.3765  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=65: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.6235 MB).
task 65: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 65
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  30 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  30 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  30 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  30 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  30 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  30 2973 0         FOF_PList    380.5744   18861.3881  fof_fof()/fof.c/249
  30 2974 0         FOF_GList    634.2907   19495.6789  fof_fof()/fof.c/337
  30 2975 0             Group     12.9689   19508.6478  fof_fof()/fof.c/380
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  47 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  47 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  47 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  47 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  47 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  47 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  47 2973 0         FOF_PList    380.5829   18861.3966  fof_fof()/fof.c/249
  47 2974 0         FOF_GList    634.3048   19495.7013  fof_fof()/fof.c/337
  47 2975 0             Group     10.4154   19506.1167  fof_fof()/fof.c/380
  47 2976 0   DomainNodeIndex      0.3469   19506.4637  force_treeallocate()/forcetree.c/4800
  47 2977 0        Nodes_base   3737.5296   23243.9933  force_treeallocate()/forcetree.c/4803
  47 2978 0     Extnodes_base   3114.6080   26358.6013  force_treeallocate()/forcetree.c/4811
  47 2979 0          Nextnode    222.3021   26580.9033  force_treeallocate()/forcetree.c/4820
  30 2976 0   DomainNodeIndex      0.3469   19508.9947  force_treeallocate()/forcetree.c/4800
  30 2977 0        Nodes_base   3737.5296   23246.5243  force_treeallocate()/forcetree.c/4803
  80 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  80 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  80 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  80 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  80 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  80 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  80 2973 0         FOF_PList    380.4036   18861.2173  fof_fof()/fof.c/249
  80 2974 0         FOF_GList    634.0060   19495.2234  fof_fof()/fof.c/337
  80 2975 0             Group     11.0950   19506.3184  fof_fof()/fof.c/380
  80 2976 0   DomainNodeIndex      0.3469   19506.6653  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  47 2980 0            Father    221.9056   26802.8089  force_treeallocate()/forcetree.c/4827
  47 2981 0           Ngblist     63.4305   26866.2394  subfind_density()/subfind_density.c/78
  47 2982 0         Dist2list    126.8610   26993.1004  subfind_density()/subfind_density.c/79
  47 2983 0    DataIndexTable     21.8182   27014.9185  subfind_density()/subfind_density.c/87
  47 2984 0      DataNodeList     58.1818   27073.1003  subfind_density()/subfind_density.c/89
  30 2978 0     Extnodes_base   3114.6080   26361.1323  force_treeallocate()/forcetree.c/4811
  30 2979 0          Nextnode    222.3021   26583.4344  force_treeallocate()/forcetree.c/4820
  80 2977 0        Nodes_base   3737.5296   23244.1949  force_treeallocate()/forcetree.c/4803
  80 2978 0     Extnodes_base   3114.6080   26358.8029  force_treeallocate()/forcetree.c/4811
  80 2979 0          Nextnode    222.3021   26581.1050  force_treeallocate()/forcetree.c/4820
  71 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  71 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  71 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  71 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  71 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  71 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  71 2973 0         FOF_PList    380.4918   18861.3055  fof_fof()/fof.c/249
  71 2974 0         FOF_GList    634.1531   19495.4586  fof_fof()/fof.c/337
  71 2975 0             Group     10.4154   19505.8740  fof_fof()/fof.c/380
  71 2976 0   DomainNodeIndex      0.3469   19506.2209  force_treeallocate()/forcetree.c/4800
  71 2977 0        Nodes_base   3737.5296   23243.7505  force_treeallocate()/forcetree.c/4803
  47 2985 0              Left    126.8610   27199.9613  subfind_density()/subfind_density.c/91
  47 2986 0             Right    126.8610   27326.8222  subfind_density()/subfind_density.c/92
  47 2987 0              Todo     15.8576   27342.6799  subfind_density()/subfind_density.c/93
  30 2980 0            Father    221.9056   26805.3400  force_treeallocate()/forcetree.c/4827
  30 2981 0           Ngblist     63.4291   26868.7691  subfind_density()/subfind_density.c/78
  30 2982 0         Dist2list    126.8581   26995.6272  subfind_density()/subfind_density.c/79
  80 2980 0            Father    221.9056   26803.0106  force_treeallocate()/forcetree.c/4827
  80 2981 0           Ngblist     63.4006   26866.4112  subfind_density()/subfind_density.c/78
  71 2978 0     Extnodes_base   3114.6080   26358.3585  force_treeallocate()/forcetree.c/4811
  71 2979 0          Nextnode    222.3021   26580.6606  force_treeallocate()/forcetree.c/4820
  71 2980 0            Father    221.9056   26802.5662  force_treeallocate()/forcetree.c/4827
  71 2981 0           Ngblist     63.4153   26865.9815  subfind_density()/subfind_density.c/78
  47 2988 0             DM_Vx    126.8610   27469.5408  subfind_density()/subfind_density.c/95
  47 2989 0             DM_Vy    126.8610   27596.4018  subfind_density()/subfind_density.c/96
  47 2990 0             DM_Vz    126.8610   27723.2627  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2983 0    DataIndexTable     21.8182   27017.4454  subfind_density()/subfind_density.c/87
  30 2984 0      DataNodeList     58.1818   27075.6272  subfind_density()/subfind_density.c/89
  30 2985 0              Left    126.8581   27202.4853  subfind_density()/subfind_density.c/91
  80 2982 0         Dist2list    126.8012   26993.2124  subfind_density()/subfind_density.c/79
  80 2983 0    DataIndexTable     21.8182   27015.0306  subfind_density()/subfind_density.c/87
  80 2984 0      DataNodeList     58.1818   27073.2124  subfind_density()/subfind_density.c/89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  71 2982 0         Dist2list    126.8306   26992.8121  subfind_density()/subfind_density.c/79
  71 2983 0    DataIndexTable     21.8182   27014.6303  subfind_density()/subfind_density.c/87
  71 2984 0      DataNodeList     58.1818   27072.8121  subfind_density()/subfind_density.c/89
  71 2985 0              Left    126.8306   27199.6427  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  47 2991 0    	  DensDataGet    109.0983   27832.3610  subfind_density()/subfind_density.c/161
  47 2992 0  	  DensDataResult     81.8237   27914.1847  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  60 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  60 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  60 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  60 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  60 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  60 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  60 2973 0         FOF_PList    380.5161   18861.3298  fof_fof()/fof.c/249
  60 2974 0         FOF_GList    634.1936   19495.5234  fof_fof()/fof.c/337
  60 2975 0             Group     11.5319   19507.0553  fof_fof()/fof.c/380
  60 2976 0   DomainNodeIndex      0.3469   19507.4022  force_treeallocate()/forcetree.c/4800
  30 2986 0             Right    126.8581   27329.3435  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  80 2985 0              Left    126.8012   27200.0136  subfind_density()/subfind_density.c/91
  80 2986 0             Right    126.8012   27326.8148  subfind_density()/subfind_density.c/92
  80 2987 0              Todo     15.8502   27342.6649  subfind_density()/subfind_density.c/93
  80 2988 0             DM_Vx    126.8012   27469.4661  subfind_density()/subfind_density.c/95
  80 2989 0             DM_Vy    126.8012   27596.2674  subfind_density()/subfind_density.c/96
  80 2990 0             DM_Vz    126.8012   27723.0686  subfind_density()/subfind_density.c/97
  80 2991 0    	  DensDataGet    121.1709   27844.2395  subfind_density()/subfind_density.c/161
  67 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  67 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  67 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  67 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  67 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  67 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  67 2973 0         FOF_PList    380.2856   18861.0993  fof_fof()/fof.c/249
  67 2974 0         FOF_GList    633.8094   19494.9087  fof_fof()/fof.c/337
  67 2975 0             Group     10.4154   19505.3241  fof_fof()/fof.c/380
  67 2976 0   DomainNodeIndex      0.3469   19505.6711  force_treeallocate()/forcetree.c/4800
  67 2977 0        Nodes_base   3737.5296   23243.2007  force_treeallocate()/forcetree.c/4803
  67 2978 0     Extnodes_base   3114.6080   26357.8087  force_treeallocate()/forcetree.c/4811
  67 2979 0          Nextnode    222.3021   26580.1107  force_treeallocate()/forcetree.c/4820
  67 2980 0            Father    221.9056   26802.0163  force_treeallocate()/forcetree.c/4827
  67 2981 0           Ngblist     63.3809   26865.3973  subfind_density()/subfind_density.c/78
  67 2982 0         Dist2list    126.7619   26992.1591  subfind_density()/subfind_density.c/79
  67 2983 0    DataIndexTable     21.8182   27013.9773  subfind_density()/subfind_density.c/87
  67 2984 0      DataNodeList     58.1818   27072.1591  subfind_density()/subfind_density.c/89
  67 2985 0              Left    126.7619   27198.9210  subfind_density()/subfind_density.c/91
  71 2986 0             Right    126.8306   27326.4733  subfind_density()/subfind_density.c/92
  71 2987 0              Todo     15.8538   27342.3271  subfind_density()/subfind_density.c/93
  71 2988 0             DM_Vx    126.8306   27469.1577  subfind_density()/subfind_density.c/95
  50 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  50 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  50 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  50 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  50 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  50 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570

Task=47: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.8153 MB).
task 47: endrun called with an error level of 812


  60 2977 0        Nodes_base   3737.5296   23244.9318  force_treeallocate()/forcetree.c/4803
  60 2978 0     Extnodes_base   3114.6080   26359.5398  force_treeallocate()/forcetree.c/4811
  30 2987 0              Todo     15.8573   27345.2008  subfind_density()/subfind_density.c/93
  77 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  77 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  77 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  77 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  77 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  77 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  77 2973 0         FOF_PList    380.4664   18861.2801  fof_fof()/fof.c/249
  77 2974 0         FOF_GList    634.1106   19495.3907  fof_fof()/fof.c/337
  77 2975 0             Group     12.0465   19507.4372  fof_fof()/fof.c/380
  77 2976 0   DomainNodeIndex      0.3469   19507.7841  force_treeallocate()/forcetree.c/4800
  77 2977 0        Nodes_base   3737.5296   23245.3137  force_treeallocate()/forcetree.c/4803
  77 2978 0     Extnodes_base   3114.6080   26359.9217  force_treeallocate()/forcetree.c/4811
  77 2979 0          Nextnode    222.3021   26582.2238  force_treeallocate()/forcetree.c/4820
  77 2980 0            Father    221.9056   26804.1294  force_treeallocate()/forcetree.c/4827
  80 2992 0  	  DensDataResult     90.8782   27935.1176  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  67 2986 0             Right    126.7619   27325.6829  subfind_density()/subfind_density.c/92
  67 2987 0              Todo     15.8452   27341.5281  subfind_density()/subfind_density.c/93
  67 2988 0             DM_Vx    126.7619   27468.2900  subfind_density()/subfind_density.c/95
  67 2989 0             DM_Vy    126.7619   27595.0519  subfind_density()/subfind_density.c/96
  67 2990 0             DM_Vz    126.7619   27721.8138  subfind_density()/subfind_density.c/97
  71 2989 0             DM_Vy    126.8306   27595.9884  subfind_density()/subfind_density.c/96
  71 2990 0             DM_Vz    126.8306   27722.8190  subfind_density()/subfind_density.c/97
  71 2991 0    	  DensDataGet    139.9799   27862.7989  subfind_density()/subfind_density.c/161
  50 2973 0         FOF_PList    380.5527   18861.3664  fof_fof()/fof.c/249
  50 2974 0         FOF_GList    634.2545   19495.6209  fof_fof()/fof.c/337
  50 2975 0             Group     10.6458   19506.2666  fof_fof()/fof.c/380
  50 2976 0   DomainNodeIndex      0.3469   19506.6136  force_treeallocate()/forcetree.c/4800
  50 2977 0        Nodes_base   3737.5296   23244.1432  force_treeallocate()/forcetree.c/4803
  50 2978 0     Extnodes_base   3114.6080   26358.7512  force_treeallocate()/forcetree.c/4811
  50 2979 0          Nextnode    222.3021   26581.0532  force_treeallocate()/forcetree.c/4820
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 47
  60 2979 0          Nextnode    222.3021   26581.8419  force_treeallocate()/forcetree.c/4820
  60 2980 0            Father    221.9056   26803.7475  force_treeallocate()/forcetree.c/4827
  60 2981 0           Ngblist     63.4194   26867.1668  subfind_density()/subfind_density.c/78
  30 2988 0             DM_Vx    126.8581   27472.0589  subfind_density()/subfind_density.c/95
  77 2981 0           Ngblist     63.4111   26867.5404  subfind_density()/subfind_density.c/78
  77 2982 0         Dist2list    126.8221   26994.3625  subfind_density()/subfind_density.c/79
  77 2983 0    DataIndexTable     21.8182   27016.1807  subfind_density()/subfind_density.c/87
  77 2984 0      DataNodeList     58.1818   27074.3625  subfind_density()/subfind_density.c/89

Task=80: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.8824 MB).
  67 2991 0    	  DensDataGet    116.3486   27838.1623  subfind_density()/subfind_density.c/161
  67 2992 0  	  DensDataResult     87.2614   27925.4238  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  71 2992 0  	  DensDataResult    104.9849   27967.7838  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=71: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=32.2162 MB).
task 71: endrun called with an error level of 812


  50 2980 0            Father    221.9056   26802.9588  force_treeallocate()/forcetree.c/4827
  50 2981 0           Ngblist     63.4255   26866.3843  subfind_density()/subfind_density.c/78
  50 2982 0         Dist2list    126.8509   26993.2352  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  60 2982 0         Dist2list    126.8387   26994.0055  subfind_density()/subfind_density.c/79
  60 2983 0    DataIndexTable     21.8182   27015.8237  subfind_density()/subfind_density.c/87
  60 2984 0      DataNodeList     58.1818   27074.0055  subfind_density()/subfind_density.c/89
  30 2989 0             DM_Vy    126.8581   27598.9170  subfind_density()/subfind_density.c/96
  77 2985 0              Left    126.8221   27201.1846  subfind_density()/subfind_density.c/91
  77 2986 0             Right    126.8221   27328.0068  subfind_density()/subfind_density.c/92
  77 2987 0              Todo     15.8528   27343.8595  subfind_density()/subfind_density.c/93
task 80: endrun called with an error level of 812



Task=67: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.5762 MB).
task 67: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 71
  50 2983 0    DataIndexTable     21.8182   27015.0534  subfind_density()/subfind_density.c/87
  50 2984 0      DataNodeList     58.1818   27073.2352  subfind_density()/subfind_density.c/89
  50 2985 0              Left    126.8509   27200.0861  subfind_density()/subfind_density.c/91
  46 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  46 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  46 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  46 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  46 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  46 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  46 2973 0         FOF_PList    379.8550   18860.6687  fof_fof()/fof.c/249
  60 2985 0              Left    126.8387   27200.8442  subfind_density()/subfind_density.c/91
  60 2986 0             Right    126.8387   27327.6829  subfind_density()/subfind_density.c/92
  30 2990 0             DM_Vz    126.8581   27725.7752  subfind_density()/subfind_density.c/97
  77 2988 0             DM_Vx    126.8221   27470.6816  subfind_density()/subfind_density.c/95
  77 2989 0             DM_Vy    126.8221   27597.5038  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 80
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 67
  50 2986 0             Right    126.8509   27326.9370  subfind_density()/subfind_density.c/92
  50 2987 0              Todo     15.8564   27342.7933  subfind_density()/subfind_density.c/93
  50 2988 0             DM_Vx    126.8509   27469.6442  subfind_density()/subfind_density.c/95
  46 2974 0         FOF_GList    633.0917   19493.7603  fof_fof()/fof.c/337
  46 2975 0             Group     12.9503   19506.7106  fof_fof()/fof.c/380
  46 2976 0   DomainNodeIndex      0.3469   19507.0575  force_treeallocate()/forcetree.c/4800
  60 2987 0              Todo     15.8548   27343.5378  subfind_density()/subfind_density.c/93
  60 2988 0             DM_Vx    126.8387   27470.3765  subfind_density()/subfind_density.c/95
  30 2991 0    	  DensDataGet    108.8633   27834.6385  subfind_density()/subfind_density.c/161
  30 2992 0  	  DensDataResult     81.6475   27916.2859  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=30: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.7141 MB).
  77 2990 0             DM_Vz    126.8221   27724.3259  subfind_density()/subfind_density.c/97
  77 2991 0    	  DensDataGet    123.1304   27847.4563  subfind_density()/subfind_density.c/161
  77 2992 0  	  DensDataResult     92.3478   27939.8040  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------- An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 80:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
  50 2989 0             DM_Vy    126.8509   27596.4951  subfind_density()/subfind_density.c/96
  50 2990 0             DM_Vz    126.8509   27723.3460  subfind_density()/subfind_density.c/97
  46 2977 0        Nodes_base   3737.5296   23244.5871  force_treeallocate()/forcetree.c/4803
  46 2978 0     Extnodes_base   3114.6080   26359.1951  force_treeallocate()/forcetree.c/4811
  46 2979 0          Nextnode    222.3021   26581.4972  force_treeallocate()/forcetree.c/4820
  60 2989 0             DM_Vy    126.8387   27597.2152  subfind_density()/subfind_density.c/96
task 30: endrun called with an error level of 812



task 80: endrun called with an error level of 124


  50 2991 0    	  DensDataGet    144.3314   27867.6774  subfind_density()/subfind_density.c/161
  50 2992 0  	  DensDataResult    108.2486   27975.9260  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=50: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=24.074 MB).
  46 2980 0            Father    221.9056   26803.4028  force_treeallocate()/forcetree.c/4827
  46 2981 0           Ngblist     63.3092   26866.7120  subfind_density()/subfind_density.c/78
  46 2982 0         Dist2list    126.6183   26993.3303  subfind_density()/subfind_density.c/79
  60 2990 0             DM_Vz    126.8387   27724.0539  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 30

Task=77: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.196 MB).
task 77: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 80
task 50: endrun called with an error level of 812


  46 2983 0    DataIndexTable     21.8182   27015.1485  subfind_density()/subfind_density.c/87
  46 2984 0      DataNodeList     58.1818   27073.3303  subfind_density()/subfind_density.c/89
  46 2985 0              Left    126.6183   27199.9486  subfind_density()/subfind_density.c/91
  60 2991 0    	  DensDataGet    128.7461   27852.8000  subfind_density()/subfind_density.c/161
  60 2992 0  	  DensDataResult     96.5596   27949.3596  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 77
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 50
  46 2986 0             Right    126.6183   27326.5669  subfind_density()/subfind_density.c/92

Task=60: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.6404 MB).
task 60: endrun called with an error level of 812


  81 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  81 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  81 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  81 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  81 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  81 2973 0         FOF_PList    380.5535   18861.3672  fof_fof()/fof.c/249
  81 2974 0         FOF_GList    634.2559   19495.6231  fof_fof()/fof.c/337
  81 2975 0             Group     10.4154   19506.0385  fof_fof()/fof.c/380
  81 2976 0   DomainNodeIndex      0.3469   19506.3855  force_treeallocate()/forcetree.c/4800
  81 2977 0        Nodes_base   3737.5296   23243.9151  force_treeallocate()/forcetree.c/4803
  46 2987 0              Todo     15.8273   27342.3942  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 60
  81 2978 0     Extnodes_base   3114.6080   26358.5231  force_treeallocate()/forcetree.c/4811
  81 2979 0          Nextnode    222.3021   26580.8252  force_treeallocate()/forcetree.c/4820
  81 2980 0            Father    221.9056   26802.7307  force_treeallocate()/forcetree.c/4827
  81 2981 0           Ngblist     63.4256   26866.1563  subfind_density()/subfind_density.c/78
  46 2988 0             DM_Vx    126.6183   27469.0126  subfind_density()/subfind_density.c/95
  81 2982 0         Dist2list    126.8512   26993.0075  subfind_density()/subfind_density.c/79
  81 2983 0    DataIndexTable     21.8182   27014.8257  subfind_density()/subfind_density.c/87
  46 2989 0             DM_Vy    126.6183   27595.6309  subfind_density()/subfind_density.c/96
  81 2984 0      DataNodeList     58.1818   27073.0075  subfind_density()/subfind_density.c/89
  46 2990 0             DM_Vz    126.6183   27722.2492  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  81 2985 0              Left    126.8512   27199.8587  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  46 2991 0    	  DensDataGet    115.1857   27837.4350  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  76 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  76 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  76 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  76 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2986 0             Right    126.8512   27326.7098  subfind_density()/subfind_density.c/92
  93 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  93 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  93 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  93 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  93 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  93 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  93 2973 0         FOF_PList    380.5761   18861.3898  fof_fof()/fof.c/249
  93 2974 0         FOF_GList    634.2934   19495.6832  fof_fof()/fof.c/337
  93 2975 0             Group     11.2289   19506.9121  fof_fof()/fof.c/380
  93 2976 0   DomainNodeIndex      0.3469   19507.2590  force_treeallocate()/forcetree.c/4800
  93 2977 0        Nodes_base   3737.5296   23244.7886  force_treeallocate()/forcetree.c/4803
  93 2978 0     Extnodes_base   3114.6080   26359.3966  force_treeallocate()/forcetree.c/4811
  93 2979 0          Nextnode    222.3021   26581.6987  force_treeallocate()/forcetree.c/4820
  93 2980 0            Father    221.9056   26803.6043  force_treeallocate()/forcetree.c/4827
  93 2981 0           Ngblist     63.4293   26867.0336  subfind_density()/subfind_density.c/78
  93 2982 0         Dist2list    126.8587   26993.8923  subfind_density()/subfind_density.c/79
  93 2983 0    DataIndexTable     21.8182   27015.7105  subfind_density()/subfind_density.c/87
  93 2984 0      DataNodeList     58.1818   27073.8923  subfind_density()/subfind_density.c/89
  51 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  51 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  51 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  51 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  51 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  51 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  51 2973 0         FOF_PList    380.5770   18861.3907  fof_fof()/fof.c/249
  51 2974 0         FOF_GList    634.2950   19495.6856  fof_fof()/fof.c/337
  51 2975 0             Group     12.2457   19507.9314  fof_fof()/fof.c/380
  51 2976 0   DomainNodeIndex      0.3469   19508.2783  force_treeallocate()/forcetree.c/4800
  51 2977 0        Nodes_base   3737.5296   23245.8079  force_treeallocate()/forcetree.c/4803
  51 2978 0     Extnodes_base   3114.6080   26360.4159  force_treeallocate()/forcetree.c/4811
  51 2979 0          Nextnode    222.3021   26582.7180  force_treeallocate()/forcetree.c/4820
  51 2980 0            Father    221.9056   26804.6236  force_treeallocate()/forcetree.c/4827
  46 2992 0  	  DensDataResult     86.3893   27923.8242  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  34 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  34 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  34 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  34 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  34 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  34 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  34 2973 0         FOF_PList    380.5188   18861.3325  fof_fof()/fof.c/249
  34 2974 0         FOF_GList    634.1980   19495.5304  fof_fof()/fof.c/337
  34 2975 0             Group     10.7993   19506.3297  fof_fof()/fof.c/380
  34 2976 0   DomainNodeIndex      0.3469   19506.6767  force_treeallocate()/forcetree.c/4800
  34 2977 0        Nodes_base   3737.5296   23244.2063  force_treeallocate()/forcetree.c/4803
  34 2978 0     Extnodes_base   3114.6080   26358.8143  force_treeallocate()/forcetree.c/4811
  34 2979 0          Nextnode    222.3021   26581.1164  force_treeallocate()/forcetree.c/4820
  34 2980 0            Father    221.9056   26803.0219  force_treeallocate()/forcetree.c/4827
  76 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  76 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  81 2987 0              Todo     15.8564   27342.5662  subfind_density()/subfind_density.c/93
  93 2985 0              Left    126.8587   27200.7510  subfind_density()/subfind_density.c/91
  93 2986 0             Right    126.8587   27327.6097  subfind_density()/subfind_density.c/92
  93 2987 0              Todo     15.8573   27343.4670  subfind_density()/subfind_density.c/93
  93 2988 0             DM_Vx    126.8587   27470.3257  subfind_density()/subfind_density.c/95
  93 2989 0             DM_Vy    126.8587   27597.1844  subfind_density()/subfind_density.c/96
  93 2990 0             DM_Vz    126.8587   27724.0431  subfind_density()/subfind_density.c/97
  51 2981 0           Ngblist     63.4295   26868.0531  subfind_density()/subfind_density.c/78
  51 2982 0         Dist2list    126.8590   26994.9121  subfind_density()/subfind_density.c/79
  51 2983 0    DataIndexTable     21.8182   27016.7302  subfind_density()/subfind_density.c/87
  51 2984 0      DataNodeList     58.1818   27074.9120  subfind_density()/subfind_density.c/89

  34 2981 0           Ngblist     63.4198   26866.4417  subfind_density()/subfind_density.c/78
  76 2973 0         FOF_PList    380.3007   18861.1144  fof_fof()/fof.c/249
  76 2974 0         FOF_GList    633.8345   19494.9489  fof_fof()/fof.c/337
  81 2988 0             DM_Vx    126.8512   27469.4174  subfind_density()/subfind_density.c/95
  81 2989 0             DM_Vy    126.8512   27596.2686  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  93 2991 0    	  DensDataGet    120.2887   27844.3318  subfind_density()/subfind_density.c/161
  93 2992 0  	  DensDataResult     90.2165   27934.5483  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  51 2985 0              Left    126.8590   27201.7710  subfind_density()/subfind_density.c/91
  51 2986 0             Right    126.8590   27328.6300  subfind_density()/subfind_density.c/92
  51 2987 0              Todo     15.8574   27344.4874  subfind_density()/subfind_density.c/93

Task=46: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.1758 MB).
  34 2982 0         Dist2list    126.8396   26993.2813  subfind_density()/subfind_density.c/79
  34 2983 0    DataIndexTable     21.8182   27015.0995  subfind_density()/subfind_density.c/87
  34 2984 0      DataNodeList     58.1818   27073.2813  subfind_density()/subfind_density.c/89
  76 2975 0             Group     11.8317   19506.7806  fof_fof()/fof.c/380
  76 2976 0   DomainNodeIndex      0.3469   19507.1275  force_treeallocate()/forcetree.c/4800
  81 2990 0             DM_Vz    126.8512   27723.1198  subfind_density()/subfind_density.c/97
  99 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  99 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  99 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  99 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  99 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  99 2973 0         FOF_PList    380.4854   18861.2991  fof_fof()/fof.c/249
  99 2974 0         FOF_GList    634.1423   19495.4414  fof_fof()/fof.c/337
  99 2975 0             Group     10.4154   19505.8568  fof_fof()/fof.c/380
  99 2976 0   DomainNodeIndex      0.3469   19506.2038  force_treeallocate()/forcetree.c/4800
  28 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  28 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  28 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  28 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  28 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  28 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  28 2973 0         FOF_PList    380.0560   18860.8697  fof_fof()/fof.c/249
  28 2974 0         FOF_GList    633.4267   19494.2964  fof_fof()/fof.c/337
  28 2975 0             Group     12.5082   19506.8047  fof_fof()/fof.c/380

Task=93: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.4517 MB).
task 93: endrun called with an error level of 812


  51 2988 0             DM_Vx    126.8590   27471.3464  subfind_density()/subfind_density.c/95
  51 2989 0             DM_Vy    126.8590   27598.2054  subfind_density()/subfind_density.c/96
  51 2990 0             DM_Vz    126.8590   27725.0644  subfind_density()/subfind_density.c/97
  51 2991 0    	  DensDataGet    123.7137   27848.7781  subfind_density()/subfind_density.c/161
task 46: endrun called with an error level of 812


  34 2985 0              Left    126.8396   27200.1209  subfind_density()/subfind_density.c/91
  34 2986 0             Right    126.8396   27326.9605  subfind_density()/subfind_density.c/92
  34 2987 0              Todo     15.8549   27342.8154  subfind_density()/subfind_density.c/93
  76 2977 0        Nodes_base   3737.5296   23244.6571  force_treeallocate()/forcetree.c/4803
  76 2978 0     Extnodes_base   3114.6080   26359.2651  force_treeallocate()/forcetree.c/4811
  76 2979 0          Nextnode    222.3021   26581.5672  force_treeallocate()/forcetree.c/4820
  81 2991 0    	  DensDataGet    136.5471   27859.6668  subfind_density()/subfind_density.c/161
  99 2977 0        Nodes_base   3737.5296   23243.7334  force_treeallocate()/forcetree.c/4803
  99 2978 0     Extnodes_base   3114.6080   26358.3414  force_treeallocate()/forcetree.c/4811
  99 2979 0          Nextnode    222.3021   26580.6435  force_treeallocate()/forcetree.c/4820
  99 2980 0            Father    221.9056   26802.5490  force_treeallocate()/forcetree.c/4827
  28 2976 0   DomainNodeIndex      0.3469   19507.1516  force_treeallocate()/forcetree.c/4800
  28 2977 0        Nodes_base   3737.5296   23244.6812  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 93
  51 2992 0  	  DensDataResult     92.7853   27941.5634  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=51: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.4366 MB).
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 46
  34 2988 0             DM_Vx    126.8396   27469.6550  subfind_density()/subfind_density.c/95
  34 2989 0             DM_Vy    126.8396   27596.4946  subfind_density()/subfind_density.c/96
  34 2990 0             DM_Vz    126.8396   27723.3342  subfind_density()/subfind_density.c/97
  76 2980 0            Father    221.9056   26803.4728  force_treeallocate()/forcetree.c/4827
  76 2981 0           Ngblist     63.3835   26866.8563  subfind_density()/subfind_density.c/78
  81 2992 0  	  DensDataResult    102.4103   27962.0771  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  99 2981 0           Ngblist     63.4142   26865.9633  subfind_density()/subfind_density.c/78
  99 2982 0         Dist2list    126.8285   26992.7917  subfind_density()/subfind_density.c/79
  99 2983 0    DataIndexTable     21.8182   27014.6099  subfind_density()/subfind_density.c/87
  99 2984 0      DataNodeList     58.1818   27072.7917  subfind_density()/subfind_density.c/89
  28 2978 0     Extnodes_base   3114.6080   26359.2892  force_treeallocate()/forcetree.c/4811
  28 2979 0          Nextnode    222.3021   26581.5913  force_treeallocate()/forcetree.c/4820
task 51: endrun called with an error level of 812


  34 2991 0    	  DensDataGet    113.0436   27836.3778  subfind_density()/subfind_density.c/161
  76 2982 0         Dist2list    126.7669   26993.6232  subfind_density()/subfind_density.c/79

  99 2985 0              Left    126.8285   27199.6202  subfind_density()/subfind_density.c/91
  99 2986 0             Right    126.8285   27326.4487  subfind_density()/subfind_density.c/92
  99 2987 0              Todo     15.8536   27342.3022  subfind_density()/subfind_density.c/93
  28 2980 0            Father    221.9056   26803.4969  force_treeallocate()/forcetree.c/4827
  28 2981 0           Ngblist     63.3427   26866.8395  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 51
  34 2992 0  	  DensDataResult     84.7827   27921.1605  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  76 2983 0    DataIndexTable     21.8182   27015.4413  subfind_density()/subfind_density.c/87

Task=81: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=37.9229 MB).
task 81: endrun called with an error level of 812


  99 2988 0             DM_Vx    126.8285   27469.1307  subfind_density()/subfind_density.c/95
  99 2989 0             DM_Vy    126.8285   27595.9592  subfind_density()/subfind_density.c/96
  99 2990 0             DM_Vz    126.8285   27722.7876  subfind_density()/subfind_density.c/97
  28 2982 0         Dist2list    126.6853   26993.5249  subfind_density()/subfind_density.c/79
  28 2983 0    DataIndexTable     21.8182   27015.3430  subfind_density()/subfind_density.c/87

Task=34: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.8395 MB).
task 34: endrun called with an error level of 812


  76 2984 0      DataNodeList     58.1818   27073.6231  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 81
  99 2991 0    	  DensDataGet    116.6257   27839.4133  subfind_density()/subfind_density.c/161
  99 2992 0  	  DensDataResult     87.4693   27926.8825  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  28 2984 0      DataNodeList     58.1818   27073.5248  subfind_density()/subfind_density.c/89
  28 2985 0              Left    126.6853   27200.2102  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 34
  76 2985 0              Left    126.7669   27200.3900  subfind_density()/subfind_density.c/91

  28 2986 0             Right    126.6853   27326.8955  subfind_density()/subfind_density.c/92
  76 2986 0             Right    126.7669   27327.1569  subfind_density()/subfind_density.c/92

Task=99: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.1175 MB).
task 99: endrun called with an error level of 812


  28 2987 0              Todo     15.8357   27342.7312  subfind_density()/subfind_density.c/93
  76 2987 0              Todo     15.8459   27343.0028  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 99
  28 2988 0             DM_Vx    126.6853   27469.4165  subfind_density()/subfind_density.c/95
  76 2988 0             DM_Vx    126.7669   27469.7697  subfind_density()/subfind_density.c/95
  28 2989 0             DM_Vy    126.6853   27596.1019  subfind_density()/subfind_density.c/96
  76 2989 0             DM_Vy    126.7669   27596.5366  subfind_density()/subfind_density.c/96
  28 2990 0             DM_Vz    126.6853   27722.7872  subfind_density()/subfind_density.c/97
  76 2990 0             DM_Vz    126.7669   27723.3035  subfind_density()/subfind_density.c/97
  28 2991 0    	  DensDataGet    118.4030   27841.1902  subfind_density()/subfind_density.c/161
  76 2991 0    	  DensDataGet    128.8264   27852.1298  subfind_density()/subfind_density.c/161
  28 2992 0  	  DensDataResult     88.8023   27929.9925  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  76 2992 0  	  DensDataResult     96.6198   27948.7496  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
Task=28: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.0075 MB).
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

task 28: endrun called with an error level of 812


  49 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  49 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  49 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  49 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  49 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  49 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  49 2973 0         FOF_PList    380.5842   18861.3979  fof_fof()/fof.c/249
  49 2974 0         FOF_GList    634.3070   19495.7049  fof_fof()/fof.c/337
  49 2975 0             Group     12.4854   19508.1903  fof_fof()/fof.c/380
  49 2976 0   DomainNodeIndex      0.3469   19508.5372  force_treeallocate()/forcetree.c/4800
  49 2977 0        Nodes_base   3737.5296   23246.0668  force_treeallocate()/forcetree.c/4803
  49 2978 0     Extnodes_base   3114.6080   26360.6748  force_treeallocate()/forcetree.c/4811
  49 2979 0          Nextnode    222.3021   26582.9769  force_treeallocate()/forcetree.c/4820
  49 2980 0            Father    221.9056   26804.8825  force_treeallocate()/forcetree.c/4827
  49 2981 0           Ngblist     63.4307   26868.3132  subfind_density()/subfind_density.c/78

Task=76: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.2504 MB).
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 28
  49 2982 0         Dist2list    126.8614   26995.1746  subfind_density()/subfind_density.c/79
  49 2983 0    DataIndexTable     21.8182   27016.9928  subfind_density()/subfind_density.c/87
  49 2984 0      DataNodeList     58.1818   27075.1745  subfind_density()/subfind_density.c/89
  49 2985 0              Left    126.8614   27202.0359  subfind_density()/subfind_density.c/91
task 76: endrun called with an error level of 812


  49 2986 0             Right    126.8614   27328.8973  subfind_density()/subfind_density.c/92
  49 2987 0              Todo     15.8577   27344.7550  subfind_density()/subfind_density.c/93
  49 2988 0             DM_Vx    126.8614   27471.6164  subfind_density()/subfind_density.c/95
  49 2989 0             DM_Vy    126.8614   27598.4778  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 76
  49 2990 0             DM_Vz    126.8614   27725.3392  subfind_density()/subfind_density.c/97
  49 2991 0    	  DensDataGet    126.7485   27852.0877  subfind_density()/subfind_density.c/161
  49 2992 0  	  DensDataResult     95.0614   27947.1490  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=49: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.851 MB).
task 49: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 49
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  89 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  89 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  89 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  89 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  89 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  89 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  89 2973 0         FOF_PList    380.5804   18861.3941  fof_fof()/fof.c/249
  89 2974 0         FOF_GList    634.3007   19495.6948  fof_fof()/fof.c/337
  89 2975 0             Group     11.4810   19507.1758  fof_fof()/fof.c/380
  89 2976 0   DomainNodeIndex      0.3469   19507.5227  force_treeallocate()/forcetree.c/4800
  89 2977 0        Nodes_base   3737.5296   23245.0523  force_treeallocate()/forcetree.c/4803
  89 2978 0     Extnodes_base   3114.6080   26359.6603  force_treeallocate()/forcetree.c/4811
  89 2979 0          Nextnode    222.3021   26581.9624  force_treeallocate()/forcetree.c/4820
  89 2980 0            Father    221.9056   26803.8680  force_treeallocate()/forcetree.c/4827
  89 2981 0           Ngblist     63.4301   26867.2981  subfind_density()/subfind_density.c/78
  89 2982 0         Dist2list    126.8601   26994.1582  subfind_density()/subfind_density.c/79
  89 2983 0    DataIndexTable     21.8182   27015.9764  subfind_density()/subfind_density.c/87
  89 2984 0      DataNodeList     58.1818   27074.1582  subfind_density()/subfind_density.c/89
  89 2985 0              Left    126.8601   27201.0183  subfind_density()/subfind_density.c/91
  89 2986 0             Right    126.8601   27327.8785  subfind_density()/subfind_density.c/92
 112 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 112 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 112 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 112 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 112 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 112 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 112 2973 0         FOF_PList    380.5752   18861.3889  fof_fof()/fof.c/249
 112 2974 0         FOF_GList    634.2921   19495.6810  fof_fof()/fof.c/337
 112 2975 0             Group     11.6948   19507.3758  fof_fof()/fof.c/380
 112 2976 0   DomainNodeIndex      0.3469   19507.7227  force_treeallocate()/forcetree.c/4800
 112 2977 0        Nodes_base   3737.5296   23245.2523  force_treeallocate()/forcetree.c/4803
 112 2978 0     Extnodes_base   3114.6080   26359.8603  force_treeallocate()/forcetree.c/4811
  89 2987 0              Todo     15.8575   27343.7360  subfind_density()/subfind_density.c/93
  89 2988 0             DM_Vx    126.8601   27470.5961  subfind_density()/subfind_density.c/95
  89 2989 0             DM_Vy    126.8601   27597.4563  subfind_density()/subfind_density.c/96
  89 2990 0             DM_Vz    126.8601   27724.3164  subfind_density()/subfind_density.c/97
  89 2991 0    	  DensDataGet    114.4133   27838.7297  subfind_density()/subfind_density.c/161
  89 2992 0  	  DensDataResult     85.8100   27924.5397  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 112 2979 0          Nextnode    222.3021   26582.1624  force_treeallocate()/forcetree.c/4820
 112 2980 0            Father    221.9056   26804.0680  force_treeallocate()/forcetree.c/4827

Task=89: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.4603 MB).
task 89: endrun called with an error level of 812


 112 2981 0           Ngblist     63.4292   26867.4972  subfind_density()/subfind_density.c/78
 112 2982 0         Dist2list    126.8584   26994.3556  subfind_density()/subfind_density.c/79
 112 2983 0    DataIndexTable     21.8182   27016.1738  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 89
 112 2984 0      DataNodeList     58.1818   27074.3556  subfind_density()/subfind_density.c/89
 112 2985 0              Left    126.8584   27201.2140  subfind_density()/subfind_density.c/91
 112 2986 0             Right    126.8584   27328.0724  subfind_density()/subfind_density.c/92
 112 2987 0              Todo     15.8573   27343.9297  subfind_density()/subfind_density.c/93
 112 2988 0             DM_Vx    126.8584   27470.7881  subfind_density()/subfind_density.c/95
 112 2989 0             DM_Vy    126.8584   27597.6465  subfind_density()/subfind_density.c/96
 112 2990 0             DM_Vz    126.8584   27724.5049  subfind_density()/subfind_density.c/97
 112 2991 0    	  DensDataGet    119.1785   27843.6835  subfind_density()/subfind_density.c/161
 112 2992 0  	  DensDataResult     89.3839   27933.0674  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=112: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.9326 MB).
task 112: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 112
task 112: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 112:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 112
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  48 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  48 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  48 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  48 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  48 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  79 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  79 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  79 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  79 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  79 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  79 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  79 2973 0         FOF_PList    380.4542   18861.2679  fof_fof()/fof.c/249
  79 2974 0         FOF_GList    634.0903   19495.3581  fof_fof()/fof.c/337
  79 2975 0             Group     10.4694   19505.8275  fof_fof()/fof.c/380
  79 2976 0   DomainNodeIndex      0.3469   19506.1744  force_treeallocate()/forcetree.c/4800
  79 2977 0        Nodes_base   3737.5296   23243.7040  force_treeallocate()/forcetree.c/4803
  79 2978 0     Extnodes_base   3114.6080   26358.3120  force_treeallocate()/forcetree.c/4811
  79 2979 0          Nextnode    222.3021   26580.6141  force_treeallocate()/forcetree.c/4820
  79 2980 0            Father    221.9056   26802.5197  force_treeallocate()/forcetree.c/4827
  79 2981 0           Ngblist     63.4090   26865.9287  subfind_density()/subfind_density.c/78
  79 2982 0         Dist2list    126.8181   26992.7468  subfind_density()/subfind_density.c/79
  79 2983 0    DataIndexTable     21.8182   27014.5649  subfind_density()/subfind_density.c/87
  79 2984 0      DataNodeList     58.1818   27072.7467  subfind_density()/subfind_density.c/89
  48 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  48 2973 0         FOF_PList    380.5088   18861.3224  fof_fof()/fof.c/249
  48 2974 0         FOF_GList    634.1813   19495.5037  fof_fof()/fof.c/337
  79 2985 0              Left    126.8181   27199.5648  subfind_density()/subfind_density.c/91
  79 2986 0             Right    126.8181   27326.3828  subfind_density()/subfind_density.c/92
  79 2987 0              Todo     15.8523   27342.2351  subfind_density()/subfind_density.c/93
  79 2988 0             DM_Vx    126.8181   27469.0532  subfind_density()/subfind_density.c/95
  79 2989 0             DM_Vy    126.8181   27595.8712  subfind_density()/subfind_density.c/96
  79 2990 0             DM_Vz    126.8181   27722.6893  subfind_density()/subfind_density.c/97
  48 2975 0             Group     12.1824   19507.6861  fof_fof()/fof.c/380
  48 2976 0   DomainNodeIndex      0.3469   19508.0331  force_treeallocate()/forcetree.c/4800
  48 2977 0        Nodes_base   3737.5296   23245.5627  force_treeallocate()/forcetree.c/4803
  79 2991 0    	  DensDataGet    109.2968   27831.9860  subfind_density()/subfind_density.c/161
  79 2992 0  	  DensDataResult     81.9726   27913.9586  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=79: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.0414 MB).
  48 2978 0     Extnodes_base   3114.6080   26360.1707  force_treeallocate()/forcetree.c/4811
  48 2979 0          Nextnode    222.3021   26582.4727  force_treeallocate()/forcetree.c/4820
  48 2980 0            Father    221.9056   26804.3783  force_treeallocate()/forcetree.c/4827
task 79: endrun called with an error level of 812


  48 2981 0           Ngblist     63.4181   26867.7965  subfind_density()/subfind_density.c/78
  48 2982 0         Dist2list    126.8363   26994.6327  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  48 2983 0    DataIndexTable     21.8182   27016.4509  subfind_density()/subfind_density.c/87
  48 2984 0      DataNodeList     58.1818   27074.6327  subfind_density()/subfind_density.c/89
  48 2985 0              Left    126.8363   27201.4689  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  37 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  37 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  37 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  37 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  37 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  37 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  37 2973 0         FOF_PList    380.2178   18861.0314  fof_fof()/fof.c/249
  37 2974 0         FOF_GList    633.6963   19494.7277  fof_fof()/fof.c/337
  37 2975 0             Group     10.8751   19505.6028  fof_fof()/fof.c/380
  37 2976 0   DomainNodeIndex      0.3469   19505.9497  force_treeallocate()/forcetree.c/4800
  37 2977 0        Nodes_base   3737.5296   23243.4793  force_treeallocate()/forcetree.c/4803
  37 2978 0     Extnodes_base   3114.6080   26358.0873  force_treeallocate()/forcetree.c/4811
  37 2979 0          Nextnode    222.3021   26580.3894  force_treeallocate()/forcetree.c/4820
  37 2980 0            Father    221.9056   26802.2950  force_treeallocate()/forcetree.c/4827
  90 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  90 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  90 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  90 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  90 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  90 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  90 2973 0         FOF_PList    380.5538   18861.3675  fof_fof()/fof.c/249
  90 2974 0         FOF_GList    634.2563   19495.6238  fof_fof()/fof.c/337
  90 2975 0             Group     11.5516   19507.1754  fof_fof()/fof.c/380
  48 2986 0             Right    126.8363   27328.3052  subfind_density()/subfind_density.c/92
 103 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 103 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 103 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 103 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 103 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 103 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 103 2973 0         FOF_PList    380.5235   18861.3372  fof_fof()/fof.c/249
 103 2974 0         FOF_GList    634.2059   19495.5431  fof_fof()/fof.c/337
 103 2975 0             Group     11.9946   19507.5378  fof_fof()/fof.c/380
 103 2976 0   DomainNodeIndex      0.3469   19507.8847  force_treeallocate()/forcetree.c/4800
 103 2977 0        Nodes_base   3737.5296   23245.4143  force_treeallocate()/forcetree.c/4803
 103 2978 0     Extnodes_base   3114.6080   26360.0223  force_treeallocate()/forcetree.c/4811
 103 2979 0          Nextnode    222.3021   26582.3244  force_treeallocate()/forcetree.c/4820
 103 2980 0            Father    221.9056   26804.2299  force_treeallocate()/forcetree.c/4827
 103 2981 0           Ngblist     63.4206   26867.6505  subfind_density()/subfind_density.c/78
 103 2982 0         Dist2list    126.8412   26994.4917  subfind_density()/subfind_density.c/79
 103 2983 0    DataIndexTable     21.8182   27016.3099  subfind_density()/subfind_density.c/87
 103 2984 0      DataNodeList     58.1818   27074.4917  subfind_density()/subfind_density.c/89
 103 2985 0              Left    126.8412   27201.3329  subfind_density()/subfind_density.c/91
 103 2986 0             Right    126.8412   27328.1740  subfind_density()/subfind_density.c/92
  37 2981 0           Ngblist     63.3696   26865.6646  subfind_density()/subfind_density.c/78
  37 2982 0         Dist2list    126.7393   26992.4038  subfind_density()/subfind_density.c/79
  37 2983 0    DataIndexTable     21.8182   27014.2220  subfind_density()/subfind_density.c/87
  37 2984 0      DataNodeList     58.1818   27072.4038  subfind_density()/subfind_density.c/89
  37 2985 0              Left    126.7393   27199.1431  subfind_density()/subfind_density.c/91
  90 2976 0   DomainNodeIndex      0.3469   19507.5223  force_treeallocate()/forcetree.c/4800
  90 2977 0        Nodes_base   3737.5296   23245.0519  force_treeallocate()/forcetree.c/4803
  90 2978 0     Extnodes_base   3114.6080   26359.6599  force_treeallocate()/forcetree.c/4811
  48 2987 0              Todo     15.8545   27344.1597  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 103 2987 0              Todo     15.8551   27344.0292  subfind_density()/subfind_density.c/93
 103 2988 0             DM_Vx    126.8412   27470.8704  subfind_density()/subfind_density.c/95
 103 2989 0             DM_Vy    126.8412   27597.7116  subfind_density()/subfind_density.c/96
 103 2990 0             DM_Vz    126.8412   27724.5527  subfind_density()/subfind_density.c/97
 103 2991 0    	  DensDataGet    135.1672   27859.7200  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  37 2986 0             Right    126.7393   27325.8823  subfind_density()/subfind_density.c/92
  37 2987 0              Todo     15.8424   27341.7247  subfind_density()/subfind_density.c/93
  37 2988 0             DM_Vx    126.7393   27468.4640  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  90 2979 0          Nextnode    222.3021   26581.9620  force_treeallocate()/forcetree.c/4820
  90 2980 0            Father    221.9056   26803.8676  force_treeallocate()/forcetree.c/4827
  90 2981 0           Ngblist     63.4256   26867.2932  subfind_density()/subfind_density.c/78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  48 2988 0             DM_Vx    126.8363   27470.9960  subfind_density()/subfind_density.c/95
  96 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  96 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  96 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  96 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  96 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  96 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  96 2973 0         FOF_PList    380.4846   18861.2982  fof_fof()/fof.c/249
  96 2974 0         FOF_GList    634.1409   19495.4392  fof_fof()/fof.c/337
  96 2975 0             Group     11.8680   19507.3072  fof_fof()/fof.c/380
 103 2992 0  	  DensDataResult    101.3754   27961.0954  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=103: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=38.9046 MB).
task 103: endrun called with an error level of 812


  61 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  61 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  61 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  61 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  61 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  61 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  61 2973 0         FOF_PList    380.5800   18861.3937  fof_fof()/fof.c/249
  61 2974 0         FOF_GList    634.3000   19495.6938  fof_fof()/fof.c/337
  61 2975 0             Group     11.9843   19507.6780  fof_fof()/fof.c/380
  61 2976 0   DomainNodeIndex      0.3469   19508.0249  force_treeallocate()/forcetree.c/4800
  61 2977 0        Nodes_base   3737.5296   23245.5545  force_treeallocate()/forcetree.c/4803
  61 2978 0     Extnodes_base   3114.6080   26360.1625  force_treeallocate()/forcetree.c/4811
  61 2979 0          Nextnode    222.3021   26582.4646  force_treeallocate()/forcetree.c/4820
  61 2980 0            Father    221.9056   26804.3702  force_treeallocate()/forcetree.c/4827
  37 2989 0             DM_Vy    126.7393   27595.2032  subfind_density()/subfind_density.c/96
  37 2990 0             DM_Vz    126.7393   27721.9425  subfind_density()/subfind_density.c/97
  37 2991 0    	  DensDataGet    133.5075   27855.4500  subfind_density()/subfind_density.c/161
  31 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  31 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  31 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  31 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  31 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  31 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  31 2973 0         FOF_PList    380.4860   18861.2997  fof_fof()/fof.c/249
  31 2974 0         FOF_GList    634.1434   19495.4431  fof_fof()/fof.c/337
  31 2975 0             Group     12.5373   19507.9804  fof_fof()/fof.c/380
  31 2976 0   DomainNodeIndex      0.3469   19508.3273  force_treeallocate()/forcetree.c/4800
  31 2977 0        Nodes_base   3737.5296   23245.8569  force_treeallocate()/forcetree.c/4803
  31 2978 0     Extnodes_base   3114.6080   26360.4649  force_treeallocate()/forcetree.c/4811
  31 2979 0          Nextnode    222.3021   26582.7670  force_treeallocate()/forcetree.c/4820
  31 2980 0            Father    221.9056   26804.6726  force_treeallocate()/forcetree.c/4827
  31 2981 0           Ngblist     63.4143   26868.0869  subfind_density()/subfind_density.c/78
  90 2982 0         Dist2list    126.8513   26994.1445  subfind_density()/subfind_density.c/79
  90 2983 0    DataIndexTable     21.8182   27015.9627  subfind_density()/subfind_density.c/87
  90 2984 0      DataNodeList     58.1818   27074.1445  subfind_density()/subfind_density.c/89
  66 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  66 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  66 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  66 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  48 2989 0             DM_Vy    126.8363   27597.8322  subfind_density()/subfind_density.c/96
  48 2990 0             DM_Vz    126.8363   27724.6685  subfind_density()/subfind_density.c/97
  96 2976 0   DomainNodeIndex      0.3469   19507.6542  force_treeallocate()/forcetree.c/4800
  96 2977 0        Nodes_base   3737.5296   23245.1838  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 103
  61 2981 0           Ngblist     63.4300   26867.8002  subfind_density()/subfind_density.c/78
  61 2982 0         Dist2list    126.8600   26994.6602  subfind_density()/subfind_density.c/79
  61 2983 0    DataIndexTable     21.8182   27016.4784  subfind_density()/subfind_density.c/87
  61 2984 0      DataNodeList     58.1818   27074.6602  subfind_density()/subfind_density.c/89
  37 2992 0  	  DensDataResult    100.1306   27955.5806  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=37: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=44.4194 MB).
task 37: endrun called with an error level of 812


  31 2982 0         Dist2list    126.8287   26994.9156  subfind_density()/subfind_density.c/79
  31 2983 0    DataIndexTable     21.8182   27016.7338  subfind_density()/subfind_density.c/87
  31 2984 0      DataNodeList     58.1818   27074.9156  subfind_density()/subfind_density.c/89
  31 2985 0              Left    126.8287   27201.7442  subfind_density()/subfind_density.c/91
  90 2985 0              Left    126.8513   27200.9957  subfind_density()/subfind_density.c/91
  90 2986 0             Right    126.8513   27327.8470  subfind_density()/subfind_density.c/92
  90 2987 0              Todo     15.8564   27343.7034  subfind_density()/subfind_density.c/93
  66 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  66 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  48 2991 0    	  DensDataGet    124.0374   27848.7059  subfind_density()/subfind_density.c/161
  96 2978 0     Extnodes_base   3114.6080   26359.7918  force_treeallocate()/forcetree.c/4811
  96 2979 0          Nextnode    222.3021   26582.0938  force_treeallocate()/forcetree.c/4820
  96 2980 0            Father    221.9056   26803.9994  force_treeallocate()/forcetree.c/4827
  61 2985 0              Left    126.8600   27201.5202  subfind_density()/subfind_density.c/91
  61 2986 0             Right    126.8600   27328.3802  subfind_density()/subfind_density.c/92
  61 2987 0              Todo     15.8575   27344.2377  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 37
  31 2986 0             Right    126.8287   27328.5729  subfind_density()/subfind_density.c/92
  31 2987 0              Todo     15.8536   27344.4265  subfind_density()/subfind_density.c/93
  31 2988 0             DM_Vx    126.8287   27471.2552  subfind_density()/subfind_density.c/95
  31 2989 0             DM_Vy    126.8287   27598.0838  subfind_density()/subfind_density.c/96
  90 2988 0             DM_Vx    126.8513   27470.5547  subfind_density()/subfind_density.c/95
  66 2973 0         FOF_PList    380.1535   18860.9671  fof_fof()/fof.c/249
  66 2974 0         FOF_GList    633.5891   19494.5562  fof_fof()/fof.c/337
  48 2992 0  	  DensDataResult     93.0281   27941.7339  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  96 2981 0           Ngblist     63.4141   26867.4135  subfind_density()/subfind_density.c/78
  96 2982 0         Dist2list    126.8282   26994.2417  subfind_density()/subfind_density.c/79
  61 2988 0             DM_Vx    126.8600   27471.0977  subfind_density()/subfind_density.c/95
  61 2989 0             DM_Vy    126.8600   27597.9577  subfind_density()/subfind_density.c/96
  61 2990 0             DM_Vz    126.8600   27724.8177  subfind_density()/subfind_density.c/97
  61 2991 0    	  DensDataGet    125.1116   27849.9294  subfind_density()/subfind_density.c/161
  31 2990 0             DM_Vz    126.8287   27724.9125  subfind_density()/subfind_density.c/97
  31 2991 0    	  DensDataGet    115.5826   27840.4952  subfind_density()/subfind_density.c/161
  31 2992 0  	  DensDataResult     86.6870   27927.1821  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  90 2989 0             DM_Vy    126.8513   27597.4059  subfind_density()/subfind_density.c/96
  66 2975 0             Group     12.6224   19507.1786  fof_fof()/fof.c/380
  66 2976 0   DomainNodeIndex      0.3469   19507.5256  force_treeallocate()/forcetree.c/4800

  96 2983 0    DataIndexTable     21.8182   27016.0599  subfind_density()/subfind_density.c/87
  96 2984 0      DataNodeList     58.1818   27074.2417  subfind_density()/subfind_density.c/89
  96 2985 0              Left    126.8282   27201.0699  subfind_density()/subfind_density.c/91
  61 2992 0  	  DensDataResult     93.8337   27943.7631  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=61: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.2369 MB).
task 61: endrun called with an error level of 812



Task=31: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.8179 MB).
task 31: endrun called with an error level of 812


  90 2990 0             DM_Vz    126.8513   27724.2572  subfind_density()/subfind_density.c/97
  66 2977 0        Nodes_base   3737.5296   23245.0552  force_treeallocate()/forcetree.c/4803
  66 2978 0     Extnodes_base   3114.6080   26359.6632  force_treeallocate()/forcetree.c/4811

Task=48: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.2661 MB).
  96 2986 0             Right    126.8282   27327.8980  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 61
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 31
  90 2991 0    	  DensDataGet    107.9217   27832.1789  subfind_density()/subfind_density.c/161
  66 2979 0          Nextnode    222.3021   26581.9652  force_treeallocate()/forcetree.c/4820
  66 2980 0            Father    221.9056   26803.8708  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
task 48: endrun called with an error level of 812


  96 2987 0              Todo     15.8535   27343.7516  subfind_density()/subfind_density.c/93
  90 2992 0  	  DensDataResult     80.9413   27913.1202  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  66 2981 0           Ngblist     63.3589   26867.2297  subfind_density()/subfind_density.c/78
  41 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  41 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  41 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  41 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  41 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  41 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  41 2973 0         FOF_PList    380.5847   18861.3984  fof_fof()/fof.c/249
  41 2974 0         FOF_GList    634.3078   19495.7062  fof_fof()/fof.c/337
  41 2975 0             Group     11.4976   19507.2038  fof_fof()/fof.c/380
  41 2976 0   DomainNodeIndex      0.3469   19507.5508  force_treeallocate()/forcetree.c/4800
  41 2977 0        Nodes_base   3737.5296   23245.0804  force_treeallocate()/forcetree.c/4803
  41 2978 0     Extnodes_base   3114.6080   26359.6884  force_treeallocate()/forcetree.c/4811
  41 2979 0          Nextnode    222.3021   26581.9904  force_treeallocate()/forcetree.c/4820
  41 2980 0            Father    221.9056   26803.8960  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 48
  96 2988 0             DM_Vx    126.8282   27470.5798  subfind_density()/subfind_density.c/95

Task=90: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.8798 MB).
task 90: endrun called with an error level of 812


  86 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  86 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  86 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  86 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  86 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  86 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  86 2973 0         FOF_PList    380.4709   18861.2846  fof_fof()/fof.c/249
  86 2974 0         FOF_GList    634.1182   19495.4029  fof_fof()/fof.c/337
  66 2982 0         Dist2list    126.7178   26993.9475  subfind_density()/subfind_density.c/79
  41 2981 0           Ngblist     63.4308   26867.3268  subfind_density()/subfind_density.c/78
  41 2982 0         Dist2list    126.8616   26994.1884  subfind_density()/subfind_density.c/79
  41 2983 0    DataIndexTable     21.8182   27016.0066  subfind_density()/subfind_density.c/87
  41 2984 0      DataNodeList     58.1818   27074.1883  subfind_density()/subfind_density.c/89
 An HDF5 error was detected, we'd better stop.
  96 2989 0             DM_Vy    126.8282   27597.4079  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 90
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  86 2975 0             Group     11.3876   19506.7905  fof_fof()/fof.c/380
  86 2976 0   DomainNodeIndex      0.3469   19507.1374  force_treeallocate()/forcetree.c/4800
  66 2983 0    DataIndexTable     21.8182   27015.7657  subfind_density()/subfind_density.c/87
  41 2985 0              Left    126.8616   27201.0499  subfind_density()/subfind_density.c/91
  41 2986 0             Right    126.8616   27327.9115  subfind_density()/subfind_density.c/92
  41 2987 0              Todo     15.8577   27343.7692  subfind_density()/subfind_density.c/93
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 48:
  96 2990 0             DM_Vz    126.8282   27724.2361  subfind_density()/subfind_density.c/97
  19 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  19 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  19 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  19 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  19 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  19 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  19 2973 0         FOF_PList    380.5763   18861.3900  fof_fof()/fof.c/249
  19 2974 0         FOF_GList    634.2939   19495.6839  fof_fof()/fof.c/337
  19 2975 0             Group     12.9285   19508.6124  fof_fof()/fof.c/380
  19 2976 0   DomainNodeIndex      0.3469   19508.9593  force_treeallocate()/forcetree.c/4800
  19 2977 0        Nodes_base   3737.5296   23246.4889  force_treeallocate()/forcetree.c/4803
  19 2978 0     Extnodes_base   3114.6080   26361.0969  force_treeallocate()/forcetree.c/4811
  19 2979 0          Nextnode    222.3021   26583.3990  force_treeallocate()/forcetree.c/4820
  19 2980 0            Father    221.9056   26805.3046  force_treeallocate()/forcetree.c/4827
  86 2977 0        Nodes_base   3737.5296   23244.6670  force_treeallocate()/forcetree.c/4803
  86 2978 0     Extnodes_base   3114.6080   26359.2750  force_treeallocate()/forcetree.c/4811
  86 2979 0          Nextnode    222.3021   26581.5771  force_treeallocate()/forcetree.c/4820
  66 2984 0      DataNodeList     58.1818   27073.9475  subfind_density()/subfind_density.c/89
  41 2988 0             DM_Vx    126.8616   27470.6307  subfind_density()/subfind_density.c/95
  41 2989 0             DM_Vy    126.8616   27597.4923  subfind_density()/subfind_density.c/96
task 48: endrun called with an error level of 124


  96 2991 0    	  DensDataGet    124.5190   27848.7551  subfind_density()/subfind_density.c/161
  19 2981 0           Ngblist     63.4294   26868.7340  subfind_density()/subfind_density.c/78
  19 2982 0         Dist2list    126.8588   26995.5928  subfind_density()/subfind_density.c/79
  19 2983 0    DataIndexTable     21.8182   27017.4109  subfind_density()/subfind_density.c/87
  19 2984 0      DataNodeList     58.1818   27075.5927  subfind_density()/subfind_density.c/89
  19 2985 0              Left    126.8588   27202.4515  subfind_density()/subfind_density.c/91
  86 2980 0            Father    221.9056   26803.4827  force_treeallocate()/forcetree.c/4827
  86 2981 0           Ngblist     63.4118   26866.8945  subfind_density()/subfind_density.c/78
  86 2982 0         Dist2list    126.8236   26993.7182  subfind_density()/subfind_density.c/79
  86 2983 0    DataIndexTable     21.8182   27015.5363  subfind_density()/subfind_density.c/87
  86 2984 0      DataNodeList     58.1818   27073.7181  subfind_density()/subfind_density.c/89
  66 2985 0              Left    126.7178   27200.6653  subfind_density()/subfind_density.c/91
  41 2990 0             DM_Vz    126.8616   27724.3539  subfind_density()/subfind_density.c/97
  41 2991 0    	  DensDataGet    135.8302   27860.1841  subfind_density()/subfind_density.c/161
  41 2992 0  	  DensDataResult    101.8727   27962.0567  subfind_density()/subfind_density.c/201
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 48
  96 2992 0  	  DensDataResult     93.3892   27942.1443  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  19 2986 0             Right    126.8588   27329.3103  subfind_density()/subfind_density.c/92
  19 2987 0              Todo     15.8574   27345.1676  subfind_density()/subfind_density.c/93
  19 2988 0             DM_Vx    126.8588   27472.0264  subfind_density()/subfind_density.c/95
  86 2985 0              Left    126.8236   27200.5418  subfind_density()/subfind_density.c/91
  86 2986 0             Right    126.8236   27327.3654  subfind_density()/subfind_density.c/92
  86 2987 0              Todo     15.8530   27343.2184  subfind_density()/subfind_density.c/93
  66 2986 0             Right    126.7178   27327.3832  subfind_density()/subfind_density.c/92
----------------------------------------------------------------------------------------

Task=96: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.8557 MB).
task 96: endrun called with an error level of 812


  19 2989 0             DM_Vy    126.8588   27598.8852  subfind_density()/subfind_density.c/96
  19 2990 0             DM_Vz    126.8588   27725.7440  subfind_density()/subfind_density.c/97
  19 2991 0    	  DensDataGet    116.1341   27841.8781  subfind_density()/subfind_density.c/161
  86 2988 0             DM_Vx    126.8236   27470.0420  subfind_density()/subfind_density.c/95
  66 2987 0              Todo     15.8397   27343.2229  subfind_density()/subfind_density.c/93

Task=41: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=37.9433 MB).
task 41: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 96
  19 2992 0  	  DensDataResult     87.1006   27928.9787  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=19: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.0213 MB).
task 19: endrun called with an error level of 812


  86 2989 0             DM_Vy    126.8236   27596.8657  subfind_density()/subfind_density.c/96
  66 2988 0             DM_Vx    126.7178   27469.9407  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 41
task 96: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 19
  86 2990 0             DM_Vz    126.8236   27723.6893  subfind_density()/subfind_density.c/97
  86 2991 0    	  DensDataGet    126.9509   27850.6402  subfind_density()/subfind_density.c/161
  66 2989 0             DM_Vy    126.7178   27596.6585  subfind_density()/subfind_density.c/96
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 96:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 96
  86 2992 0  	  DensDataResult     95.2132   27945.8533  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  66 2990 0             DM_Vz    126.7178   27723.3763  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

  66 2991 0    	  DensDataGet    139.7753   27863.1516  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  22 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  22 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  22 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  22 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  22 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  22 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  22 2973 0         FOF_PList    380.4330   18861.2467  fof_fof()/fof.c/249
  22 2974 0         FOF_GList    634.0550   19495.3017  fof_fof()/fof.c/337
  22 2975 0             Group     12.6556   19507.9572  fof_fof()/fof.c/380
  22 2976 0   DomainNodeIndex      0.3469   19508.3042  force_treeallocate()/forcetree.c/4800

Task=86: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.1467 MB).
task 86: endrun called with an error level of 812


  66 2992 0  	  DensDataResult    104.8315   27967.9831  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  39 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  39 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  39 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  39 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  39 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  39 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  39 2973 0         FOF_PList    379.8363   18860.6500  fof_fof()/fof.c/249
  39 2974 0         FOF_GList    633.0605   19493.7105  fof_fof()/fof.c/337
  39 2975 0             Group     12.4263   19506.1367  fof_fof()/fof.c/380
  39 2976 0   DomainNodeIndex      0.3469   19506.4837  force_treeallocate()/forcetree.c/4800
  39 2977 0        Nodes_base   3737.5296   23244.0133  force_treeallocate()/forcetree.c/4803
  39 2978 0     Extnodes_base   3114.6080   26358.6213  force_treeallocate()/forcetree.c/4811
  39 2979 0          Nextnode    222.3021   26580.9234  force_treeallocate()/forcetree.c/4820
  39 2980 0            Father    221.9056   26802.8289  force_treeallocate()/forcetree.c/4827
  39 2981 0           Ngblist     63.3061   26866.1350  subfind_density()/subfind_density.c/78
  39 2982 0         Dist2list    126.6121   26992.7471  subfind_density()/subfind_density.c/79
 105 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 105 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 105 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 105 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 105 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 105 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 105 2973 0         FOF_PList    380.1678   18860.9815  fof_fof()/fof.c/249
 105 2974 0         FOF_GList    633.6131   19494.5946  fof_fof()/fof.c/337
 105 2975 0             Group     11.7072   19506.3018  fof_fof()/fof.c/380
 105 2976 0   DomainNodeIndex      0.3469   19506.6487  force_treeallocate()/forcetree.c/4800
 105 2977 0        Nodes_base   3737.5296   23244.1783  force_treeallocate()/forcetree.c/4803
 105 2978 0     Extnodes_base   3114.6080   26358.7863  force_treeallocate()/forcetree.c/4811
 105 2979 0          Nextnode    222.3021   26581.0884  force_treeallocate()/forcetree.c/4820
 105 2980 0            Father    221.9056   26802.9940  force_treeallocate()/forcetree.c/4827
  22 2977 0        Nodes_base   3737.5296   23245.8338  force_treeallocate()/forcetree.c/4803
  22 2978 0     Extnodes_base   3114.6080   26360.4418  force_treeallocate()/forcetree.c/4811
  22 2979 0          Nextnode    222.3021   26582.7439  force_treeallocate()/forcetree.c/4820
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 86

  39 2983 0    DataIndexTable     21.8182   27014.5653  subfind_density()/subfind_density.c/87
  39 2984 0      DataNodeList     58.1818   27072.7471  subfind_density()/subfind_density.c/89
  39 2985 0              Left    126.6121   27199.3592  subfind_density()/subfind_density.c/91
  39 2986 0             Right    126.6121   27325.9713  subfind_density()/subfind_density.c/92
  39 2987 0              Todo     15.8265   27341.7978  subfind_density()/subfind_density.c/93
 105 2981 0           Ngblist     63.3613   26866.3553  subfind_density()/subfind_density.c/78
 105 2982 0         Dist2list    126.7226   26993.0779  subfind_density()/subfind_density.c/79
 105 2983 0    DataIndexTable     21.8182   27014.8961  subfind_density()/subfind_density.c/87
 105 2984 0      DataNodeList     58.1818   27073.0779  subfind_density()/subfind_density.c/89
 105 2985 0              Left    126.7226   27199.8005  subfind_density()/subfind_density.c/91
 105 2986 0             Right    126.7226   27326.5231  subfind_density()/subfind_density.c/92
 105 2987 0              Todo     15.8403   27342.3634  subfind_density()/subfind_density.c/93
 105 2988 0             DM_Vx    126.7226   27469.0860  subfind_density()/subfind_density.c/95
 105 2989 0             DM_Vy    126.7226   27595.8086  subfind_density()/subfind_density.c/96
  22 2980 0            Father    221.9056   26804.6494  force_treeallocate()/forcetree.c/4827
  22 2981 0           Ngblist     63.4055   26868.0549  subfind_density()/subfind_density.c/78
  22 2982 0         Dist2list    126.8110   26994.8659  subfind_density()/subfind_density.c/79

Task=66: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=32.0169 MB).
  39 2988 0             DM_Vx    126.6121   27468.4099  subfind_density()/subfind_density.c/95
  39 2989 0             DM_Vy    126.6121   27595.0220  subfind_density()/subfind_density.c/96
  39 2990 0             DM_Vz    126.6121   27721.6341  subfind_density()/subfind_density.c/97
 105 2990 0             DM_Vz    126.7226   27722.5313  subfind_density()/subfind_density.c/97
 105 2991 0    	  DensDataGet    134.3016   27856.8328  subfind_density()/subfind_density.c/161
 105 2992 0  	  DensDataResult    100.7262   27957.5590  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  22 2983 0    DataIndexTable     21.8182   27016.6841  subfind_density()/subfind_density.c/87
  22 2984 0      DataNodeList     58.1818   27074.8659  subfind_density()/subfind_density.c/89
task 66: endrun called with an error level of 812


  39 2991 0    	  DensDataGet    114.0982   27835.7323  subfind_density()/subfind_density.c/161
  39 2992 0  	  DensDataResult     85.5737   27921.3059  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=39: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.6941 MB).
task 39: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=105: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.441 MB).
task 105: endrun called with an error level of 812


  22 2985 0              Left    126.8110   27201.6769  subfind_density()/subfind_density.c/91
  22 2986 0             Right    126.8110   27328.4879  subfind_density()/subfind_density.c/92
  22 2987 0              Todo     15.8514   27344.3393  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 66
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 39
  75 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  75 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  75 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  75 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  75 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  75 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  75 2973 0         FOF_PList    380.5802   18861.3939  fof_fof()/fof.c/249
  75 2974 0         FOF_GList    634.3003   19495.6941  fof_fof()/fof.c/337
  75 2975 0             Group     10.8263   19506.5204  fof_fof()/fof.c/380
  75 2976 0   DomainNodeIndex      0.3469   19506.8673  force_treeallocate()/forcetree.c/4800
  75 2977 0        Nodes_base   3737.5296   23244.3969  force_treeallocate()/forcetree.c/4803
  75 2978 0     Extnodes_base   3114.6080   26359.0050  force_treeallocate()/forcetree.c/4811
  75 2979 0          Nextnode    222.3021   26581.3070  force_treeallocate()/forcetree.c/4820
  75 2980 0            Father    221.9056   26803.2126  force_treeallocate()/forcetree.c/4827
  75 2981 0           Ngblist     63.4300   26866.6426  subfind_density()/subfind_density.c/78
  75 2982 0         Dist2list    126.8601   26993.5027  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 105
  22 2988 0             DM_Vx    126.8110   27471.1503  subfind_density()/subfind_density.c/95
  52 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  52 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  52 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  52 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  52 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  52 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  52 2973 0         FOF_PList    380.5801   18861.3938  fof_fof()/fof.c/249
  52 2974 0         FOF_GList    634.3002   19495.6939  fof_fof()/fof.c/337
  52 2975 0             Group     12.3547   19508.0486  fof_fof()/fof.c/380
  52 2976 0   DomainNodeIndex      0.3469   19508.3955  force_treeallocate()/forcetree.c/4800
  52 2977 0        Nodes_base   3737.5296   23245.9251  force_treeallocate()/forcetree.c/4803
  75 2983 0    DataIndexTable     21.8182   27015.3209  subfind_density()/subfind_density.c/87
  75 2984 0      DataNodeList     58.1818   27073.5027  subfind_density()/subfind_density.c/89
  75 2985 0              Left    126.8601   27200.3627  subfind_density()/subfind_density.c/91
  75 2986 0             Right    126.8601   27327.2228  subfind_density()/subfind_density.c/92
  85 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  85 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  85 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  85 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  85 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  85 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  85 2973 0         FOF_PList    380.5727   18861.3863  fof_fof()/fof.c/249
  85 2974 0         FOF_GList    634.2878   19495.6741  fof_fof()/fof.c/337
  85 2975 0             Group     11.1978   19506.8719  fof_fof()/fof.c/380
  85 2976 0   DomainNodeIndex      0.3469   19507.2188  force_treeallocate()/forcetree.c/4800
  85 2977 0        Nodes_base   3737.5296   23244.7484  force_treeallocate()/forcetree.c/4803
  85 2978 0     Extnodes_base   3114.6080   26359.3564  force_treeallocate()/forcetree.c/4811
  85 2979 0          Nextnode    222.3021   26581.6585  force_treeallocate()/forcetree.c/4820
  85 2980 0            Father    221.9056   26803.5641  force_treeallocate()/forcetree.c/4827
  85 2981 0           Ngblist     63.4288   26866.9928  subfind_density()/subfind_density.c/78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  22 2989 0             DM_Vy    126.8110   27597.9613  subfind_density()/subfind_density.c/96
  52 2978 0     Extnodes_base   3114.6080   26360.5331  force_treeallocate()/forcetree.c/4811
  52 2979 0          Nextnode    222.3021   26582.8352  force_treeallocate()/forcetree.c/4820
  52 2980 0            Father    221.9056   26804.7408  force_treeallocate()/forcetree.c/4827
  75 2987 0              Todo     15.8575   27343.0803  subfind_density()/subfind_density.c/93
  75 2988 0             DM_Vx    126.8601   27469.9403  subfind_density()/subfind_density.c/95
  75 2989 0             DM_Vy    126.8601   27596.8004  subfind_density()/subfind_density.c/96
  75 2990 0             DM_Vz    126.8601   27723.6605  subfind_density()/subfind_density.c/97
  85 2982 0         Dist2list    126.8576   26993.8504  subfind_density()/subfind_density.c/79
  85 2983 0    DataIndexTable     21.8182   27015.6686  subfind_density()/subfind_density.c/87
  85 2984 0      DataNodeList     58.1818   27073.8504  subfind_density()/subfind_density.c/89
  85 2985 0              Left    126.8576   27200.7079  subfind_density()/subfind_density.c/91
 104 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 104 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 104 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 104 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 104 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 104 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 104 2973 0         FOF_PList    380.5448   18861.3584  fof_fof()/fof.c/249
  22 2990 0             DM_Vz    126.8110   27724.7723  subfind_density()/subfind_density.c/97
  52 2981 0           Ngblist     63.4300   26868.1708  subfind_density()/subfind_density.c/78
  52 2982 0         Dist2list    126.8600   26995.0309  subfind_density()/subfind_density.c/79
  75 2991 0    	  DensDataGet    131.0227   27854.6832  subfind_density()/subfind_density.c/161
  75 2992 0  	  DensDataResult     98.2670   27952.9502  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=75: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=47.0498 MB).
  85 2986 0             Right    126.8576   27327.5655  subfind_density()/subfind_density.c/92
  85 2987 0              Todo     15.8572   27343.4227  subfind_density()/subfind_density.c/93
  85 2988 0             DM_Vx    126.8576   27470.2802  subfind_density()/subfind_density.c/95
 104 2974 0         FOF_GList    634.2413   19495.5997  fof_fof()/fof.c/337
 104 2975 0             Group     10.5897   19506.1894  fof_fof()/fof.c/380
 104 2976 0   DomainNodeIndex      0.3469   19506.5364  force_treeallocate()/forcetree.c/4800
  22 2991 0    	  DensDataGet    123.4055   27848.1778  subfind_density()/subfind_density.c/161
  52 2983 0    DataIndexTable     21.8182   27016.8490  subfind_density()/subfind_density.c/87
  52 2984 0      DataNodeList     58.1818   27075.0308  subfind_density()/subfind_density.c/89
  52 2985 0              Left    126.8600   27201.8909  subfind_density()/subfind_density.c/91
task 75: endrun called with an error level of 812


  85 2989 0             DM_Vy    126.8576   27597.1378  subfind_density()/subfind_density.c/96
  85 2990 0             DM_Vz    126.8576   27723.9953  subfind_density()/subfind_density.c/97
 104 2977 0        Nodes_base   3737.5296   23244.0660  force_treeallocate()/forcetree.c/4803
 104 2978 0     Extnodes_base   3114.6080   26358.6740  force_treeallocate()/forcetree.c/4811
  22 2992 0  	  DensDataResult     92.5541   27940.7319  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  52 2986 0             Right    126.8600   27328.7509  subfind_density()/subfind_density.c/92
  52 2987 0              Todo     15.8575   27344.6084  subfind_density()/subfind_density.c/93
  52 2988 0             DM_Vx    126.8600   27471.4684  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 75
  85 2991 0    	  DensDataGet    124.6537   27848.6491  subfind_density()/subfind_density.c/161
  85 2992 0  	  DensDataResult     93.4903   27942.1394  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 104 2979 0          Nextnode    222.3021   26580.9760  force_treeallocate()/forcetree.c/4820
 104 2980 0            Father    221.9056   26802.8816  force_treeallocate()/forcetree.c/4827
 104 2981 0           Ngblist     63.4241   26866.3057  subfind_density()/subfind_density.c/78

  52 2989 0             DM_Vy    126.8600   27598.3285  subfind_density()/subfind_density.c/96
  52 2990 0             DM_Vz    126.8600   27725.1885  subfind_density()/subfind_density.c/97

Task=85: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.8606 MB).
task 85: endrun called with an error level of 812


 104 2982 0         Dist2list    126.8483   26993.1540  subfind_density()/subfind_density.c/79

Task=22: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.2681 MB).
  52 2991 0    	  DensDataGet    123.1924   27848.3809  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 85
 104 2983 0    DataIndexTable     21.8182   27014.9722  subfind_density()/subfind_density.c/87
 104 2984 0      DataNodeList     58.1818   27073.1540  subfind_density()/subfind_density.c/89
task 22: endrun called with an error level of 812


  52 2992 0  	  DensDataResult     92.3943   27940.7753  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 104 2985 0              Left    126.8483   27200.0022  subfind_density()/subfind_density.c/91
 104 2986 0             Right    126.8483   27326.8505  subfind_density()/subfind_density.c/92
 104 2987 0              Todo     15.8560   27342.7065  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 22

Task=52: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.2247 MB).
task 52: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 104 2988 0             DM_Vx    126.8483   27469.5548  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 52
  54 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  54 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  54 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  54 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  54 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  54 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  54 2973 0         FOF_PList    380.5240   18861.3377  fof_fof()/fof.c/249
  54 2974 0         FOF_GList    634.2067   19495.5444  fof_fof()/fof.c/337
  54 2975 0             Group     11.4748   19507.0192  fof_fof()/fof.c/380
  54 2976 0   DomainNodeIndex      0.3469   19507.3661  force_treeallocate()/forcetree.c/4800
 104 2989 0             DM_Vy    126.8483   27596.4030  subfind_density()/subfind_density.c/96
  54 2977 0        Nodes_base   3737.5296   23244.8957  force_treeallocate()/forcetree.c/4803
  54 2978 0     Extnodes_base   3114.6080   26359.5037  force_treeallocate()/forcetree.c/4811
 104 2990 0             DM_Vz    126.8483   27723.2513  subfind_density()/subfind_density.c/97
  54 2979 0          Nextnode    222.3021   26581.8058  force_treeallocate()/forcetree.c/4820
  54 2980 0            Father    221.9056   26803.7114  force_treeallocate()/forcetree.c/4827
  54 2981 0           Ngblist     63.4207   26867.1321  subfind_density()/subfind_density.c/78
 104 2991 0    	  DensDataGet    113.8843   27837.1355  subfind_density()/subfind_density.c/161
  54 2982 0         Dist2list    126.8413   26993.9734  subfind_density()/subfind_density.c/79
  54 2983 0    DataIndexTable     21.8182   27015.7916  subfind_density()/subfind_density.c/87
  54 2984 0      DataNodeList     58.1818   27073.9734  subfind_density()/subfind_density.c/89
  54 2985 0              Left    126.8413   27200.8147  subfind_density()/subfind_density.c/91
 104 2992 0  	  DensDataResult     85.4132   27922.5487  subfind_density()/subfind_density.c/201
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  54 2986 0             Right    126.8413   27327.6561  subfind_density()/subfind_density.c/92
  54 2987 0              Todo     15.8552   27343.5112  subfind_density()/subfind_density.c/93
  54 2988 0             DM_Vx    126.8413   27470.3526  subfind_density()/subfind_density.c/95
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  69 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  69 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  69 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  69 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  69 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  69 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  69 2973 0         FOF_PList    379.8705   18860.6842  fof_fof()/fof.c/249
  69 2974 0         FOF_GList    633.1174   19493.8016  fof_fof()/fof.c/337
  69 2975 0             Group     10.7090   19504.5107  fof_fof()/fof.c/380
  69 2976 0   DomainNodeIndex      0.3469   19504.8576  force_treeallocate()/forcetree.c/4800
  69 2977 0        Nodes_base   3737.5296   23242.3872  force_treeallocate()/forcetree.c/4803
  69 2978 0     Extnodes_base   3114.6080   26356.9952  force_treeallocate()/forcetree.c/4811
  69 2979 0          Nextnode    222.3021   26579.2973  force_treeallocate()/forcetree.c/4820
  69 2980 0            Father    221.9056   26801.2029  force_treeallocate()/forcetree.c/4827
  69 2981 0           Ngblist     63.3117   26864.5146  subfind_density()/subfind_density.c/78
  54 2989 0             DM_Vy    126.8413   27597.1939  subfind_density()/subfind_density.c/96

Task=104: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.4513 MB).
task 104: endrun called with an error level of 812


  91 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  91 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  91 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  91 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  91 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  91 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  91 2973 0         FOF_PList    380.3662   18861.1799  fof_fof()/fof.c/249
  91 2974 0         FOF_GList    633.9437   19495.1236  fof_fof()/fof.c/337
  91 2975 0             Group     12.1887   19507.3123  fof_fof()/fof.c/380
  91 2976 0   DomainNodeIndex      0.3469   19507.6592  force_treeallocate()/forcetree.c/4800
  91 2977 0        Nodes_base   3737.5296   23245.1888  force_treeallocate()/forcetree.c/4803
  91 2978 0     Extnodes_base   3114.6080   26359.7968  force_treeallocate()/forcetree.c/4811
  69 2982 0         Dist2list    126.6235   26991.1381  subfind_density()/subfind_density.c/79
  69 2983 0    DataIndexTable     21.8182   27012.9563  subfind_density()/subfind_density.c/87
  69 2984 0      DataNodeList     58.1818   27071.1381  subfind_density()/subfind_density.c/89
  69 2985 0              Left    126.6235   27197.7615  subfind_density()/subfind_density.c/91
  69 2986 0             Right    126.6235   27324.3850  subfind_density()/subfind_density.c/92
  54 2990 0             DM_Vz    126.8413   27724.0352  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 104
  91 2979 0          Nextnode    222.3021   26582.0989  force_treeallocate()/forcetree.c/4820
  91 2980 0            Father    221.9056   26804.0045  force_treeallocate()/forcetree.c/4827
  91 2981 0           Ngblist     63.3944   26867.3989  subfind_density()/subfind_density.c/78
  91 2982 0         Dist2list    126.7887   26994.1876  subfind_density()/subfind_density.c/79
  69 2987 0              Todo     15.8279   27340.2130  subfind_density()/subfind_density.c/93
  69 2988 0             DM_Vx    126.6235   27466.8365  subfind_density()/subfind_density.c/95
  69 2989 0             DM_Vy    126.6235   27593.4600  subfind_density()/subfind_density.c/96
  54 2991 0    	  DensDataGet    129.3906   27853.4258  subfind_density()/subfind_density.c/161
 An HDF5 error was detected, we'd better stop.
  91 2983 0    DataIndexTable     21.8182   27016.0058  subfind_density()/subfind_density.c/87
  91 2984 0      DataNodeList     58.1818   27074.1876  subfind_density()/subfind_density.c/89
  91 2985 0              Left    126.7887   27200.9763  subfind_density()/subfind_density.c/91
  91 2986 0             Right    126.7887   27327.7651  subfind_density()/subfind_density.c/92
  91 2987 0              Todo     15.8486   27343.6136  subfind_density()/subfind_density.c/93
  69 2990 0             DM_Vz    126.6235   27720.0834  subfind_density()/subfind_density.c/97
  69 2991 0    	  DensDataGet    115.2432   27835.3267  subfind_density()/subfind_density.c/161
  69 2992 0  	  DensDataResult     86.4324   27921.7591  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  54 2992 0  	  DensDataResult     97.0429   27950.4687  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------HDF5-DIAG: Error detected in HDF5 (1.8.12)   91 2988 0             DM_Vx    126.7887   27470.4024  subfind_density()/subfind_density.c/95
  91 2989 0             DM_Vy    126.7887   27597.1911  subfind_density()/subfind_density.c/96
  91 2990 0             DM_Vz    126.7887   27723.9799  subfind_density()/subfind_density.c/97

Task=69: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.2409 MB).
task 69: endrun called with an error level of 812



MPI-process 104:
  91 2991 0    	  DensDataGet    110.1183   27834.0982  subfind_density()/subfind_density.c/161
  91 2992 0  	  DensDataResult     82.5888   27916.6870  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 69

Task=54: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.5313 MB).
task 54: endrun called with an error level of 812


  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value

Task=91: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.313 MB).
task 91: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 54
task 104: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 91
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 104
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 109 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 109 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 109 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 109 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 109 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 109 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 109 2973 0         FOF_PList    380.3778   18861.1915  fof_fof()/fof.c/249
 109 2974 0         FOF_GList    633.9630   19495.1545  fof_fof()/fof.c/337
 109 2975 0             Group     11.4208   19506.5753  fof_fof()/fof.c/380
 109 2976 0   DomainNodeIndex      0.3469   19506.9223  force_treeallocate()/forcetree.c/4800
 109 2977 0        Nodes_base   3737.5296   23244.4519  force_treeallocate()/forcetree.c/4803
 109 2978 0     Extnodes_base   3114.6080   26359.0599  force_treeallocate()/forcetree.c/4811
 109 2979 0          Nextnode    222.3021   26581.3620  force_treeallocate()/forcetree.c/4820
 109 2980 0            Father    221.9056   26803.2675  force_treeallocate()/forcetree.c/4827
 109 2981 0           Ngblist     63.3963   26866.6638  subfind_density()/subfind_density.c/78
 109 2982 0         Dist2list    126.7926   26993.4565  subfind_density()/subfind_density.c/79
 109 2983 0    DataIndexTable     21.8182   27015.2746  subfind_density()/subfind_density.c/87
 109 2984 0      DataNodeList     58.1818   27073.4564  subfind_density()/subfind_density.c/89
 109 2985 0              Left    126.7926   27200.2490  subfind_density()/subfind_density.c/91
 109 2986 0             Right    126.7926   27327.0416  subfind_density()/subfind_density.c/92
 109 2987 0              Todo     15.8491   27342.8907  subfind_density()/subfind_density.c/93
 109 2988 0             DM_Vx    126.7926   27469.6833  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 109 2989 0             DM_Vy    126.7926   27596.4759  subfind_density()/subfind_density.c/96
 109 2990 0             DM_Vz    126.7926   27723.2685  subfind_density()/subfind_density.c/97
 109 2991 0    	  DensDataGet    125.1109   27848.3794  subfind_density()/subfind_density.c/161
  68 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  68 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  68 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  68 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  68 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 109 2992 0  	  DensDataResult     93.8332   27942.2126  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=109: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.7874 MB).
task 109: endrun called with an error level of 812


  68 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  68 2973 0         FOF_PList    380.5812   18861.3949  fof_fof()/fof.c/249
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 109
  68 2974 0         FOF_GList    634.3019   19495.6968  fof_fof()/fof.c/337
  68 2975 0             Group     12.1067   19507.8035  fof_fof()/fof.c/380
  68 2976 0   DomainNodeIndex      0.3469   19508.1504  force_treeallocate()/forcetree.c/4800
  68 2977 0        Nodes_base   3737.5296   23245.6800  force_treeallocate()/forcetree.c/4803
  68 2978 0     Extnodes_base   3114.6080   26360.2880  force_treeallocate()/forcetree.c/4811
  68 2979 0          Nextnode    222.3021   26582.5901  force_treeallocate()/forcetree.c/4820
  68 2980 0            Father    221.9056   26804.4957  force_treeallocate()/forcetree.c/4827
  68 2981 0           Ngblist     63.4302   26867.9259  subfind_density()/subfind_density.c/78
  68 2982 0         Dist2list    126.8604   26994.7863  subfind_density()/subfind_density.c/79
  68 2983 0    DataIndexTable     21.8182   27016.6045  subfind_density()/subfind_density.c/87
  68 2984 0      DataNodeList     58.1818   27074.7863  subfind_density()/subfind_density.c/89
  68 2985 0              Left    126.8604   27201.6466  subfind_density()/subfind_density.c/91
  68 2986 0             Right    126.8604   27328.5070  subfind_density()/subfind_density.c/92
  68 2987 0              Todo     15.8576   27344.3646  subfind_density()/subfind_density.c/93
  68 2988 0             DM_Vx    126.8604   27471.2250  subfind_density()/subfind_density.c/95
  68 2989 0             DM_Vy    126.8604   27598.0854  subfind_density()/subfind_density.c/96
  68 2990 0             DM_Vz    126.8604   27724.9458  subfind_density()/subfind_density.c/97
  68 2991 0    	  DensDataGet    113.4918   27838.4376  subfind_density()/subfind_density.c/161
  68 2992 0  	  DensDataResult     85.1189   27923.5564  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=68: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.4436 MB).
task 68: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  17 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  17 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  17 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  17 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  17 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  17 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  17 2973 0         FOF_PList    380.4535   18861.2671  fof_fof()/fof.c/249
  17 2974 0         FOF_GList    634.0891   19495.3562  fof_fof()/fof.c/337
  17 2975 0             Group     13.1848   19508.5410  fof_fof()/fof.c/380
  17 2976 0   DomainNodeIndex      0.3469   19508.8879  force_treeallocate()/forcetree.c/4800
  17 2977 0        Nodes_base   3737.5296   23246.4175  force_treeallocate()/forcetree.c/4803
  17 2978 0     Extnodes_base   3114.6080   26361.0255  force_treeallocate()/forcetree.c/4811
  17 2979 0          Nextnode    222.3021   26583.3276  force_treeallocate()/forcetree.c/4820
  17 2980 0            Father    221.9056   26805.2332  force_treeallocate()/forcetree.c/4827
  17 2981 0           Ngblist     63.4089   26868.6421  subfind_density()/subfind_density.c/78
  17 2982 0         Dist2list    126.8178   26995.4599  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 68
 110 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 110 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 110 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 110 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 110 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 110 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 110 2973 0         FOF_PList    380.5861   18861.3998  fof_fof()/fof.c/249
 110 2974 0         FOF_GList    634.3102   19495.7099  fof_fof()/fof.c/337
 110 2975 0             Group     11.1241   19506.8340  fof_fof()/fof.c/380
 110 2976 0   DomainNodeIndex      0.3469   19507.1809  force_treeallocate()/forcetree.c/4800
 110 2977 0        Nodes_base   3737.5296   23244.7105  force_treeallocate()/forcetree.c/4803
  17 2983 0    DataIndexTable     21.8182   27017.2781  subfind_density()/subfind_density.c/87
  17 2984 0      DataNodeList     58.1818   27075.4599  subfind_density()/subfind_density.c/89
  17 2985 0              Left    126.8178   27202.2777  subfind_density()/subfind_density.c/91
  17 2986 0             Right    126.8178   27329.0955  subfind_density()/subfind_density.c/92
 110 2978 0     Extnodes_base   3114.6080   26359.3186  force_treeallocate()/forcetree.c/4811
  17 2987 0              Todo     15.8522   27344.9478  subfind_density()/subfind_density.c/93
  17 2988 0             DM_Vx    126.8178   27471.7656  subfind_density()/subfind_density.c/95
  17 2989 0             DM_Vy    126.8178   27598.5834  subfind_density()/subfind_density.c/96
  17 2990 0             DM_Vz    126.8178   27725.4012  subfind_density()/subfind_density.c/97
 110 2979 0          Nextnode    222.3021   26581.6206  force_treeallocate()/forcetree.c/4820
 110 2980 0            Father    221.9056   26803.5262  force_treeallocate()/forcetree.c/4827
 110 2981 0           Ngblist     63.4310   26866.9572  subfind_density()/subfind_density.c/78
  17 2991 0    	  DensDataGet    118.9847   27844.3859  subfind_density()/subfind_density.c/161
  17 2992 0  	  DensDataResult     89.2386   27933.6245  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 110 2982 0         Dist2list    126.8620   26993.8193  subfind_density()/subfind_density.c/79
 110 2983 0    DataIndexTable     21.8182   27015.6374  subfind_density()/subfind_density.c/87
 110 2984 0      DataNodeList     58.1818   27073.8192  subfind_density()/subfind_density.c/89

Task=17: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.3755 MB).
task 17: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 110 2985 0              Left    126.8620   27200.6813  subfind_density()/subfind_density.c/91
 110 2986 0             Right    126.8620   27327.5433  subfind_density()/subfind_density.c/92
 110 2987 0              Todo     15.8578   27343.4010  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 17
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  87 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  87 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  87 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  87 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  87 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  87 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  87 2973 0         FOF_PList    379.7731   18860.5868  fof_fof()/fof.c/249
  87 2974 0         FOF_GList    632.9552   19493.5420  fof_fof()/fof.c/337
  87 2975 0             Group     12.1710   19505.7130  fof_fof()/fof.c/380
  87 2976 0   DomainNodeIndex      0.3469   19506.0599  force_treeallocate()/forcetree.c/4800
  87 2977 0        Nodes_base   3737.5296   23243.5895  force_treeallocate()/forcetree.c/4803
  87 2978 0     Extnodes_base   3114.6080   26358.1975  force_treeallocate()/forcetree.c/4811
  87 2979 0          Nextnode    222.3021   26580.4996  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 110 2988 0             DM_Vx    126.8620   27470.2631  subfind_density()/subfind_density.c/95
 110 2989 0             DM_Vy    126.8620   27597.1251  subfind_density()/subfind_density.c/96
  74 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  74 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  74 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  74 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  74 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  87 2980 0            Father    221.9056   26802.4052  force_treeallocate()/forcetree.c/4827
  87 2981 0           Ngblist     63.2955   26865.7007  subfind_density()/subfind_density.c/78
  87 2982 0         Dist2list    126.5910   26992.2917  subfind_density()/subfind_density.c/79
  87 2983 0    DataIndexTable     21.8182   27014.1099  subfind_density()/subfind_density.c/87
  53 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  53 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  53 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  53 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  53 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  53 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  53 2973 0         FOF_PList    380.1508   18860.9645  fof_fof()/fof.c/249
  53 2974 0         FOF_GList    633.5846   19494.5491  fof_fof()/fof.c/337
  53 2975 0             Group     11.4644   19506.0135  fof_fof()/fof.c/380
  53 2976 0   DomainNodeIndex      0.3469   19506.3605  force_treeallocate()/forcetree.c/4800
  53 2977 0        Nodes_base   3737.5296   23243.8901  force_treeallocate()/forcetree.c/4803
  53 2978 0     Extnodes_base   3114.6080   26358.4981  force_treeallocate()/forcetree.c/4811
 110 2990 0             DM_Vz    126.8620   27723.9871  subfind_density()/subfind_density.c/97
  74 2973 0         FOF_PList    379.8699   18860.6836  fof_fof()/fof.c/249
  74 2974 0         FOF_GList    633.1165   19493.8000  fof_fof()/fof.c/337
  74 2975 0             Group     12.6058   19506.4058  fof_fof()/fof.c/380
  74 2976 0   DomainNodeIndex      0.3469   19506.7527  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  87 2984 0      DataNodeList     58.1818   27072.2917  subfind_density()/subfind_density.c/89
  87 2985 0              Left    126.5910   27198.8827  subfind_density()/subfind_density.c/91
  87 2986 0             Right    126.5910   27325.4738  subfind_density()/subfind_density.c/92
  87 2987 0              Todo     15.8239   27341.2977  subfind_density()/subfind_density.c/93
  87 2988 0             DM_Vx    126.5910   27467.8887  subfind_density()/subfind_density.c/95
  53 2979 0          Nextnode    222.3021   26580.8001  force_treeallocate()/forcetree.c/4820
  53 2980 0            Father    221.9056   26802.7057  force_treeallocate()/forcetree.c/4827
  53 2981 0           Ngblist     63.3585   26866.0642  subfind_density()/subfind_density.c/78
  53 2982 0         Dist2list    126.7169   26992.7811  subfind_density()/subfind_density.c/79
 110 2991 0    	  DensDataGet    115.1846   27839.1717  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  74 2977 0        Nodes_base   3737.5296   23244.2823  force_treeallocate()/forcetree.c/4803
  74 2978 0     Extnodes_base   3114.6080   26358.8903  force_treeallocate()/forcetree.c/4811
  74 2979 0          Nextnode    222.3021   26581.1924  force_treeallocate()/forcetree.c/4820
  20 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  20 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  20 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  20 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  20 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  20 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  20 2973 0         FOF_PList    380.5430   18861.3567  fof_fof()/fof.c/249
  20 2974 0         FOF_GList    634.2383   19495.5949  fof_fof()/fof.c/337
  20 2975 0             Group     11.0930   19506.6879  fof_fof()/fof.c/380
  20 2976 0   DomainNodeIndex      0.3469   19507.0348  force_treeallocate()/forcetree.c/4800
  20 2977 0        Nodes_base   3737.5296   23244.5644  force_treeallocate()/forcetree.c/4803
  87 2989 0             DM_Vy    126.5910   27594.4797  subfind_density()/subfind_density.c/96
  87 2990 0             DM_Vz    126.5910   27721.0708  subfind_density()/subfind_density.c/97
  87 2991 0    	  DensDataGet    121.7278   27842.7985  subfind_density()/subfind_density.c/161
  53 2983 0    DataIndexTable     21.8182   27014.5993  subfind_density()/subfind_density.c/87
  53 2984 0      DataNodeList     58.1818   27072.7811  subfind_density()/subfind_density.c/89
  53 2985 0              Left    126.7169   27199.4980  subfind_density()/subfind_density.c/91
  53 2986 0             Right    126.7169   27326.2149  subfind_density()/subfind_density.c/92
 110 2992 0  	  DensDataResult     86.3884   27925.5601  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 107 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 107 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 107 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 107 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 107 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 107 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 107 2973 0         FOF_PList    380.5850   18861.3987  fof_fof()/fof.c/249
 107 2974 0         FOF_GList    634.3084   19495.7071  fof_fof()/fof.c/337
 107 2975 0             Group     11.7695   19507.4766  fof_fof()/fof.c/380
 107 2976 0   DomainNodeIndex      0.3469   19507.8235  force_treeallocate()/forcetree.c/4800
 107 2977 0        Nodes_base   3737.5296   23245.3531  force_treeallocate()/forcetree.c/4803
 107 2978 0     Extnodes_base   3114.6080   26359.9611  force_treeallocate()/forcetree.c/4811
 107 2979 0          Nextnode    222.3021   26582.2632  force_treeallocate()/forcetree.c/4820
 107 2980 0            Father    221.9056   26804.1688  force_treeallocate()/forcetree.c/4827
 107 2981 0           Ngblist     63.4308   26867.5996  subfind_density()/subfind_density.c/78
 107 2982 0         Dist2list    126.8617   26994.4613  subfind_density()/subfind_density.c/79
 107 2983 0    DataIndexTable     21.8182   27016.2795  subfind_density()/subfind_density.c/87
 107 2984 0      DataNodeList     58.1818   27074.4613  subfind_density()/subfind_density.c/89
  74 2980 0            Father    221.9056   26803.0980  force_treeallocate()/forcetree.c/4827
  74 2981 0           Ngblist     63.3116   26866.4096  subfind_density()/subfind_density.c/78
  20 2978 0     Extnodes_base   3114.6080   26359.1724  force_treeallocate()/forcetree.c/4811
  87 2992 0  	  DensDataResult     91.2958   27934.0944  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=87: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.9056 MB).
task 87: endrun called with an error level of 812


  53 2987 0              Todo     15.8396   27342.0546  subfind_density()/subfind_density.c/93
  53 2988 0             DM_Vx    126.7169   27468.7715  subfind_density()/subfind_density.c/95
  53 2989 0             DM_Vy    126.7169   27595.4884  subfind_density()/subfind_density.c/96

Task=110: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.4399 MB).
 107 2985 0              Left    126.8617   27201.3230  subfind_density()/subfind_density.c/91
 107 2986 0             Right    126.8617   27328.1846  subfind_density()/subfind_density.c/92
 107 2987 0              Todo     15.8577   27344.0424  subfind_density()/subfind_density.c/93
 107 2988 0             DM_Vx    126.8617   27470.9040  subfind_density()/subfind_density.c/95
 107 2989 0             DM_Vy    126.8617   27597.7657  subfind_density()/subfind_density.c/96
 107 2990 0             DM_Vz    126.8617   27724.6274  subfind_density()/subfind_density.c/97
 107 2991 0    	  DensDataGet    112.5180   27837.1454  subfind_density()/subfind_density.c/161
  74 2982 0         Dist2list    126.6233   26993.0329  subfind_density()/subfind_density.c/79
  74 2983 0    DataIndexTable     21.8182   27014.8511  subfind_density()/subfind_density.c/87
  74 2984 0      DataNodeList     58.1818   27073.0329  subfind_density()/subfind_density.c/89
  74 2985 0              Left    126.6233   27199.6562  subfind_density()/subfind_density.c/91
  20 2979 0          Nextnode    222.3021   26581.4745  force_treeallocate()/forcetree.c/4820
  20 2980 0            Father    221.9056   26803.3801  force_treeallocate()/forcetree.c/4827
  20 2981 0           Ngblist     63.4238   26866.8039  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 87
  53 2990 0             DM_Vz    126.7169   27722.2053  subfind_density()/subfind_density.c/97
  53 2991 0    	  DensDataGet    131.1915   27853.3969  subfind_density()/subfind_density.c/161
  53 2992 0  	  DensDataResult     98.3936   27951.7905  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 110: endrun called with an error level of 812


 107 2992 0  	  DensDataResult     84.3885   27921.5339  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=107: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.4661 MB).
task 107: endrun called with an error level of 812


  74 2986 0             Right    126.6233   27326.2795  subfind_density()/subfind_density.c/92
  74 2987 0              Todo     15.8279   27342.1074  subfind_density()/subfind_density.c/93
  20 2982 0         Dist2list    126.8477   26993.6516  subfind_density()/subfind_density.c/79
  20 2983 0    DataIndexTable     21.8182   27015.4698  subfind_density()/subfind_density.c/87

Task=53: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=48.2095 MB).
task 53: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 110
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 107
  74 2988 0             DM_Vx    126.6233   27468.7307  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  20 2984 0      DataNodeList     58.1818   27073.6516  subfind_density()/subfind_density.c/89
  20 2985 0              Left    126.8477   27200.4992  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 53
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  43 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2989 0             DM_Vy    126.6233   27595.3540  subfind_density()/subfind_density.c/96
  84 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  84 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  84 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  84 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  20 2986 0             Right    126.8477   27327.3469  subfind_density()/subfind_density.c/92
  43 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  43 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  43 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  43 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  43 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  43 2973 0         FOF_PList    380.5816   18861.3953  fof_fof()/fof.c/249
  43 2974 0         FOF_GList    634.3026   19495.6979  fof_fof()/fof.c/337
  43 2975 0             Group     11.0452   19506.7431  fof_fof()/fof.c/380
  43 2976 0   DomainNodeIndex      0.3469   19507.0901  force_treeallocate()/forcetree.c/4800
  43 2977 0        Nodes_base   3737.5296   23244.6197  force_treeallocate()/forcetree.c/4803
  43 2978 0     Extnodes_base   3114.6080   26359.2277  force_treeallocate()/forcetree.c/4811
  43 2979 0          Nextnode    222.3021   26581.5297  force_treeallocate()/forcetree.c/4820
  43 2980 0            Father    221.9056   26803.4353  force_treeallocate()/forcetree.c/4827
  43 2981 0           Ngblist     63.4303   26866.8656  subfind_density()/subfind_density.c/78
  43 2982 0         Dist2list    126.8605   26993.7261  subfind_density()/subfind_density.c/79
  43 2983 0    DataIndexTable     21.8182   27015.5443  subfind_density()/subfind_density.c/87
  43 2984 0      DataNodeList     58.1818   27073.7261  subfind_density()/subfind_density.c/89
  43 2985 0              Left    126.8605   27200.5866  subfind_density()/subfind_density.c/91
  43 2986 0             Right    126.8605   27327.4472  subfind_density()/subfind_density.c/92
  74 2990 0             DM_Vz    126.6233   27721.9773  subfind_density()/subfind_density.c/97
  84 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  84 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  20 2987 0              Todo     15.8560   27343.2028  subfind_density()/subfind_density.c/93
  43 2987 0              Todo     15.8576   27343.3047  subfind_density()/subfind_density.c/93
  43 2988 0             DM_Vx    126.8605   27470.1652  subfind_density()/subfind_density.c/95
  43 2989 0             DM_Vy    126.8605   27597.0258  subfind_density()/subfind_density.c/96
  43 2990 0             DM_Vz    126.8605   27723.8863  subfind_density()/subfind_density.c/97
  43 2991 0    	  DensDataGet    111.0490   27834.9353  subfind_density()/subfind_density.c/161
  43 2992 0  	  DensDataResult     83.2868   27918.2221  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  74 2991 0    	  DensDataGet    110.3545   27832.3318  subfind_density()/subfind_density.c/161
  84 2973 0         FOF_PList    380.4955   18861.3091  fof_fof()/fof.c/249
  84 2974 0         FOF_GList    634.1591   19495.4682  fof_fof()/fof.c/337
  84 2975 0             Group     10.4154   19505.8836  fof_fof()/fof.c/380
  20 2988 0             DM_Vx    126.8477   27470.0505  subfind_density()/subfind_density.c/95

Task=43: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.7779 MB).
task 43: endrun called with an error level of 812


  74 2992 0  	  DensDataResult     82.7659   27915.0976  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  84 2976 0   DomainNodeIndex      0.3469   19506.2306  force_treeallocate()/forcetree.c/4800
  84 2977 0        Nodes_base   3737.5296   23243.7602  force_treeallocate()/forcetree.c/4803
  20 2989 0             DM_Vy    126.8477   27596.8981  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 43

  84 2978 0     Extnodes_base   3114.6080   26358.3682  force_treeallocate()/forcetree.c/4811
  84 2979 0          Nextnode    222.3021   26580.6702  force_treeallocate()/forcetree.c/4820
  84 2980 0            Father    221.9056   26802.5758  force_treeallocate()/forcetree.c/4827
  20 2990 0             DM_Vz    126.8477   27723.7458  subfind_density()/subfind_density.c/97

Task=74: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.9024 MB).
task 74: endrun called with an error level of 812


  84 2981 0           Ngblist     63.4159   26865.9917  subfind_density()/subfind_density.c/78
  20 2991 0    	  DensDataGet    117.2506   27840.9964  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 74
  84 2982 0         Dist2list    126.8318   26992.8236  subfind_density()/subfind_density.c/79
  84 2983 0    DataIndexTable     21.8182   27014.6417  subfind_density()/subfind_density.c/87
  20 2992 0  	  DensDataResult     87.9380   27928.9344  subfind_density()/subfind_density.c/201
  84 2984 0      DataNodeList     58.1818   27072.8235  subfind_density()/subfind_density.c/89
----------------------------------------------------------------------------------------
  84 2985 0              Left    126.8318   27199.6553  subfind_density()/subfind_density.c/91

Task=20: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.0656 MB).
  84 2986 0             Right    126.8318   27326.4872  subfind_density()/subfind_density.c/92
task 20: endrun called with an error level of 812


  84 2987 0              Todo     15.8540   27342.3411  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 20
  84 2988 0             DM_Vx    126.8318   27469.1730  subfind_density()/subfind_density.c/95
  84 2989 0             DM_Vy    126.8318   27596.0048  subfind_density()/subfind_density.c/96
  84 2990 0             DM_Vz    126.8318   27722.8366  subfind_density()/subfind_density.c/97
  84 2991 0    	  DensDataGet    121.1504   27843.9870  subfind_density()/subfind_density.c/161
  84 2992 0  	  DensDataResult     90.8628   27934.8498  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=84: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.1502 MB).
task 84: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 84
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 106 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 106 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 106 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 106 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 106 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 106 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 106 2973 0         FOF_PList    380.5209   18861.3346  fof_fof()/fof.c/249
 106 2974 0         FOF_GList    634.2015   19495.5362  fof_fof()/fof.c/337
 106 2975 0             Group     10.4154   19505.9516  fof_fof()/fof.c/380
 106 2976 0   DomainNodeIndex      0.3469   19506.2985  force_treeallocate()/forcetree.c/4800
 106 2977 0        Nodes_base   3737.5296   23243.8281  force_treeallocate()/forcetree.c/4803
 106 2978 0     Extnodes_base   3114.6080   26358.4361  force_treeallocate()/forcetree.c/4811
 106 2979 0          Nextnode    222.3021   26580.7382  force_treeallocate()/forcetree.c/4820
 106 2980 0            Father    221.9056   26802.6438  force_treeallocate()/forcetree.c/4827
 106 2981 0           Ngblist     63.4202   26866.0639  subfind_density()/subfind_density.c/78
 106 2982 0         Dist2list    126.8403   26992.9042  subfind_density()/subfind_density.c/79
 106 2983 0    DataIndexTable     21.8182   27014.7224  subfind_density()/subfind_density.c/87
 106 2984 0      DataNodeList     58.1818   27072.9042  subfind_density()/subfind_density.c/89
 106 2985 0              Left    126.8403   27199.7445  subfind_density()/subfind_density.c/91
 106 2986 0             Right    126.8403   27326.5848  subfind_density()/subfind_density.c/92
 106 2987 0              Todo     15.8550   27342.4399  subfind_density()/subfind_density.c/93
 106 2988 0             DM_Vx    126.8403   27469.2802  subfind_density()/subfind_density.c/95
 106 2989 0             DM_Vy    126.8403   27596.1205  subfind_density()/subfind_density.c/96
 106 2990 0             DM_Vz    126.8403   27722.9608  subfind_density()/subfind_density.c/97
 106 2991 0    	  DensDataGet    112.6526   27835.6134  subfind_density()/subfind_density.c/161
 106 2992 0  	  DensDataResult     84.4895   27920.1029  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=106: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.8971 MB).
task 106: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 106
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 111 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 111 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 111 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 111 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 111 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
 111 2973 0         FOF_PList    380.2042   18861.0179  fof_fof()/fof.c/249
 111 2974 0         FOF_GList    633.6737   19494.6916  fof_fof()/fof.c/337
 111 2975 0             Group     11.1739   19505.8654  fof_fof()/fof.c/380
 111 2976 0   DomainNodeIndex      0.3469   19506.2124  force_treeallocate()/forcetree.c/4800
 111 2977 0        Nodes_base   3737.5296   23243.7420  force_treeallocate()/forcetree.c/4803
 111 2978 0     Extnodes_base   3114.6080   26358.3500  force_treeallocate()/forcetree.c/4811
 111 2979 0          Nextnode    222.3021   26580.6521  force_treeallocate()/forcetree.c/4820
 111 2980 0            Father    221.9056   26802.5576  force_treeallocate()/forcetree.c/4827
 111 2981 0           Ngblist     63.3674   26865.9250  subfind_density()/subfind_density.c/78
  16 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  16 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  16 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  16 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  16 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  16 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  16 2973 0         FOF_PList    380.5276   18861.3413  fof_fof()/fof.c/249
  16 2974 0         FOF_GList    634.2126   19495.5539  fof_fof()/fof.c/337
 111 2982 0         Dist2list    126.7347   26992.6598  subfind_density()/subfind_density.c/79
 111 2983 0    DataIndexTable     21.8182   27014.4779  subfind_density()/subfind_density.c/87
 111 2984 0      DataNodeList     58.1818   27072.6597  subfind_density()/subfind_density.c/89
 111 2985 0              Left    126.7347   27199.3945  subfind_density()/subfind_density.c/91
  16 2975 0             Group     13.0146   19508.5685  fof_fof()/fof.c/380
  16 2976 0   DomainNodeIndex      0.3469   19508.9154  force_treeallocate()/forcetree.c/4800
  16 2977 0        Nodes_base   3737.5296   23246.4450  force_treeallocate()/forcetree.c/4803
 111 2986 0             Right    126.7347   27326.1292  subfind_density()/subfind_density.c/92
 111 2987 0              Todo     15.8418   27341.9710  subfind_density()/subfind_density.c/93
 111 2988 0             DM_Vx    126.7347   27468.7058  subfind_density()/subfind_density.c/95
  16 2978 0     Extnodes_base   3114.6080   26361.0530  force_treeallocate()/forcetree.c/4811
  16 2979 0          Nextnode    222.3021   26583.3551  force_treeallocate()/forcetree.c/4820
  16 2980 0            Father    221.9056   26805.2607  force_treeallocate()/forcetree.c/4827
 111 2989 0             DM_Vy    126.7347   27595.4405  subfind_density()/subfind_density.c/96
 111 2990 0             DM_Vz    126.7347   27722.1752  subfind_density()/subfind_density.c/97
 111 2991 0    	  DensDataGet    133.0730   27855.2482  subfind_density()/subfind_density.c/161
  16 2981 0           Ngblist     63.4213   26868.6820  subfind_density()/subfind_density.c/78
  16 2982 0         Dist2list    126.8425   26995.5245  subfind_density()/subfind_density.c/79
  16 2983 0    DataIndexTable     21.8182   27017.3427  subfind_density()/subfind_density.c/87
 111 2992 0  	  DensDataResult     99.8047   27955.0530  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=111: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=44.947 MB).
task 111: endrun called with an error level of 812


  16 2984 0      DataNodeList     58.1818   27075.5245  subfind_density()/subfind_density.c/89
  16 2985 0              Left    126.8425   27202.3670  subfind_density()/subfind_density.c/91
  16 2986 0             Right    126.8425   27329.2095  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 111
  16 2987 0              Todo     15.8553   27345.0648  subfind_density()/subfind_density.c/93
  16 2988 0             DM_Vx    126.8425   27471.9074  subfind_density()/subfind_density.c/95
  16 2989 0             DM_Vy    126.8425   27598.7499  subfind_density()/subfind_density.c/96
  16 2990 0             DM_Vz    126.8425   27725.5924  subfind_density()/subfind_density.c/97
  16 2991 0    	  DensDataGet    124.3911   27849.9836  subfind_density()/subfind_density.c/161
  16 2992 0  	  DensDataResult     93.2933   27943.2769  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=16: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.7231 MB).
task 16: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 16
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) task 16: endrun called with an error level of 124


MPI-process 16:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 16
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  72 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  72 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  72 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  72 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  72 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  72 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  72 2973 0         FOF_PList    380.4278   18861.2415  fof_fof()/fof.c/249
  72 2974 0         FOF_GList    634.0463   19495.2878  fof_fof()/fof.c/337
  72 2975 0             Group     11.2278   19506.5156  fof_fof()/fof.c/380
  72 2976 0   DomainNodeIndex      0.3469   19506.8625  force_treeallocate()/forcetree.c/4800
  72 2977 0        Nodes_base   3737.5296   23244.3921  force_treeallocate()/forcetree.c/4803
  55 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  55 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  55 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  55 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  55 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  55 2972 0          TopNodes      5.1544   18480.8137  domain_allocate()/domain.c/570
  55 2973 0         FOF_PList    379.2859   18860.0996  fof_fof()/fof.c/249
  55 2974 0         FOF_GList    632.1432   19492.2428  fof_fof()/fof.c/337
  55 2975 0             Group     13.2999   19505.5427  fof_fof()/fof.c/380
  55 2976 0   DomainNodeIndex      0.3469   19505.8896  force_treeallocate()/forcetree.c/4800
  55 2977 0        Nodes_base   3737.5296   23243.4192  force_treeallocate()/forcetree.c/4803
  55 2978 0     Extnodes_base   3114.6080   26358.0272  force_treeallocate()/forcetree.c/4811
  55 2979 0          Nextnode    222.3021   26580.3293  force_treeallocate()/forcetree.c/4820
  72 2978 0     Extnodes_base   3114.6080   26359.0001  force_treeallocate()/forcetree.c/4811
  55 2980 0            Father    221.9056   26802.2349  force_treeallocate()/forcetree.c/4827
  55 2981 0           Ngblist     63.2143   26865.4492  subfind_density()/subfind_density.c/78
  55 2982 0         Dist2list    126.4286   26991.8778  subfind_density()/subfind_density.c/79
  72 2979 0          Nextnode    222.3021   26581.3022  force_treeallocate()/forcetree.c/4820
  72 2980 0            Father    221.9056   26803.2078  force_treeallocate()/forcetree.c/4827
  72 2981 0           Ngblist     63.4046   26866.6124  subfind_density()/subfind_density.c/78
  55 2983 0    DataIndexTable     21.8182   27013.6960  subfind_density()/subfind_density.c/87
  55 2984 0      DataNodeList     58.1818   27071.8778  subfind_density()/subfind_density.c/89
  55 2985 0              Left    126.4286   27198.3064  subfind_density()/subfind_density.c/91
  55 2986 0             Right    126.4286   27324.7351  subfind_density()/subfind_density.c/92
  72 2982 0         Dist2list    126.8093   26993.4217  subfind_density()/subfind_density.c/79
  72 2983 0    DataIndexTable     21.8182   27015.2399  subfind_density()/subfind_density.c/87
  72 2984 0      DataNodeList     58.1818   27073.4217  subfind_density()/subfind_density.c/89
  55 2987 0              Todo     15.8036   27340.5387  subfind_density()/subfind_density.c/93
  55 2988 0             DM_Vx    126.4286   27466.9673  subfind_density()/subfind_density.c/95
  55 2989 0             DM_Vy    126.4286   27593.3959  subfind_density()/subfind_density.c/96
  72 2985 0              Left    126.8093   27200.2309  subfind_density()/subfind_density.c/91
  72 2986 0             Right    126.8093   27327.0402  subfind_density()/subfind_density.c/92
  72 2987 0              Todo     15.8512   27342.8913  subfind_density()/subfind_density.c/93
  55 2990 0             DM_Vz    126.4286   27719.8246  subfind_density()/subfind_density.c/97
  55 2991 0    	  DensDataGet    121.2004   27841.0249  subfind_density()/subfind_density.c/161
  55 2992 0  	  DensDataResult     90.9003   27931.9252  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  72 2988 0             DM_Vx    126.8093   27469.7006  subfind_density()/subfind_density.c/95

Task=55: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.0748 MB).
task 55: endrun called with an error level of 812


  72 2989 0             DM_Vy    126.8093   27596.5098  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 55
  72 2990 0             DM_Vz    126.8093   27723.3191  subfind_density()/subfind_density.c/97
  72 2991 0    	  DensDataGet    122.5354   27845.8545  subfind_density()/subfind_density.c/161
  72 2992 0  	  DensDataResult     91.9016   27937.7561  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=72: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.2439 MB).
task 72: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 72
task 72: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 72:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 72
-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63110.47 Mb (on task=120), Smallest =   62841.85 Mb (on task=0), Average =   63040.03 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1689.92 Mb (on task=0), Smallest =    1421.59 Mb (on task=120), Average =    1492.05 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.11 Mb
AllocMem:	 Largest =    1689.92 Mb (on task=0), Smallest =    1421.59 Mb (on task=120), Average =    1492.05 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_023_z000p199/snap_023_z000p199.0.hdf5' on task=0 (contains 133148094 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  133035377  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      8385  (tot=     0000134208) masstab=0
Type 3 (bulge):   104332  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.834624

Setting next time for line of sight file to Time_next= 0.836046

Setting next time for snipshot file to Time_next= 0.841907

Setting next time for fof output to Time_next= 0.841907
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=9.35417 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=102321
After=102937
NTopleaves= 90070  NTopnodes=102937 (space for 465370)
gravity work-load balance=1.00036   memory-balance=1.00035   SPH work-load balance=0
iter=0 exchange of 2107199970 particles (ret=0)
domain decomposition done. (took 41.8377 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5909 MByte in top-level domain structure

Allocated 7296.17 MByte for BH-tree, and 0.34359 Mbyte for top-leaves.  (presently allocted 25777.3 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.1 Mbyte (on task=123), Smallest = 26119.3 Mbyte, Average = 26120 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 123 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 123 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 123 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 123 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 123 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 123 2972 0          TopNodes      5.1048   18480.7641  domain_allocate()/domain.c/570
 123 2973 0   DomainNodeIndex      0.3436   18481.1077  force_treeallocate()/forcetree.c/4800
 123 2974 0        Nodes_base   3737.4380   22218.5457  force_treeallocate()/forcetree.c/4803
 123 2975 0     Extnodes_base   3114.5317   25333.0774  force_treeallocate()/forcetree.c/4811
 123 2976 0          Nextnode    222.2983   25555.3757  force_treeallocate()/forcetree.c/4820
 123 2977 0            Father    221.9056   25777.2813  force_treeallocate()/forcetree.c/4827
 123 2978 0           Ngblist     63.4240   25840.7053  density()/density.c/690
 123 2979 0              Left    126.8481   25967.5534  density()/density.c/692
 123 2980 0             Right    126.8481   26094.4015  density()/density.c/693
 123 2981 0    DataIndexTable      7.0175   26101.4190  density()/density.c/738
 123 2982 0      DataNodeList     18.7134   26120.1324  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.155.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.211.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.3 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=93625
After=103585
NTopleaves= 90637  NTopnodes=103585 (space for 465370)
gravity work-load balance=1.00075   memory-balance=1.00076   SPH work-load balance=0
iter=0 exchange of 2111042273 particles (ret=0)
domain decomposition done. (took 33.801 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5612 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.8 MB)
Tree construction.

Start linking particles (presently allocated=26537.5 MB)
links on local processor done (took 158.1 sec).
Marked=0223505966 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26861.3 MB) 
have done 0002858206 cross links (processed 0223505966, took 20.6795 sec)
have done 0001265791 cross links (processed 0185254484, took 18.5602 sec)
have done 0000912506 cross links (processed 0164294145, took 18.5974 sec)
have done 0000598056 cross links (processed 0113551651, took 16.4606 sec)
have done 0000375941 cross links (processed 0068429410, took 12.5481 sec)
have done 0000230455 cross links (processed 0033060116, took 7.94408 sec)
have done 0000137926 cross links (processed 0014073815, took 3.73305 sec)
have done 0000085777 cross links (processed 0008204034, took 3.17566 sec)
have done 0000053587 cross links (processed 0009949639, took 3.25166 sec)
have done 0000027896 cross links (processed 0008751152, took 7.17922 sec)
have done 0000017752 cross links (processed 0001695556, took 0.727039 sec)
have done 0000012612 cross links (processed 0000495441, took 0.652652 sec)
have done 0000011320 cross links (processed 0000573428, took 0.70545 sec)
have done 0000008508 cross links (processed 0000237220, took 0.276981 sec)
have done 0000003208 cross links (processed 0000069465, took 0.232199 sec)
have done 0000000887 cross links (processed 0000015422, took 0.185397 sec)
have done 0000000280 cross links (processed 0000003758, took 0.178323 sec)
have done 0000000092 cross links (processed 0000001217, took 0.173853 sec)
have done 0000000022 cross links (processed 0000000387, took 0.174573 sec)
have done 0000000008 cross links (processed 0000000151, took 0.173889 sec)
have done 0000000001 cross links (processed 0000000040, took 0.177157 sec)
have done 0000000000 cross links (processed 0000000002, took 0.173525 sec)
Local groups found.

group finding took = 274.605 sec
Start finding nearest dm-particle (presently allocated=26537.5 MB)
still finding nearest... (presently allocated=26804.7 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.456972 sec
compiling local group data and catalogue took = 6.73838 sec

Total number of groups with at least 32 particles: 1306291
Largest group has 134747108 particles.
Total number of particles in groups: 1278072775

group properties are now allocated.. (presently allocated=19505.8 MB)
computation of group properties took = 0.52938 sec
start global sorting of group catalogues number 23
LargestGroup = 105040.625479; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.217498 sec
Group catalogues globally sorted. took = 7.23691 sec
starting saving of group catalogue number 23
data//groups_023_z000p199
Group catalogues number 23 saved. took = 18.0856 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 12.3632 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 2.0736 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 12.3299 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  54 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  54 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  54 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  54 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  54 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  54 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  54 2973 0         FOF_PList    380.5115   18861.3078  fof_fof()/fof.c/249
  54 2974 0         FOF_GList    634.1859   19495.4936  fof_fof()/fof.c/337
  54 2975 0             Group     13.4265   19508.9201  fof_fof()/fof.c/380
  54 2976 0   DomainNodeIndex      0.3458   19509.2659  force_treeallocate()/forcetree.c/4800
  73 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  73 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  73 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  73 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  73 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  73 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  73 2973 0         FOF_PList    380.5969   18861.3932  fof_fof()/fof.c/249
  73 2974 0         FOF_GList    634.3282   19495.7213  fof_fof()/fof.c/337
  73 2975 0             Group     11.2413   19506.9627  fof_fof()/fof.c/380
  73 2976 0   DomainNodeIndex      0.3458   19507.3084  force_treeallocate()/forcetree.c/4800
  73 2977 0        Nodes_base   3737.4974   23244.8058  force_treeallocate()/forcetree.c/4803
  73 2978 0     Extnodes_base   3114.5811   26359.3870  force_treeallocate()/forcetree.c/4811
  73 2979 0          Nextnode    222.3007   26581.6877  force_treeallocate()/forcetree.c/4820
  73 2980 0            Father    221.9056   26803.5933  force_treeallocate()/forcetree.c/4827
  73 2981 0           Ngblist     63.4328   26867.0261  subfind_density()/subfind_density.c/78
  73 2982 0         Dist2list    126.8656   26993.8917  subfind_density()/subfind_density.c/79
  54 2977 0        Nodes_base   3737.4974   23246.7633  force_treeallocate()/forcetree.c/4803
  54 2978 0     Extnodes_base   3114.5811   26361.3444  force_treeallocate()/forcetree.c/4811
  54 2979 0          Nextnode    222.3007   26583.6451  force_treeallocate()/forcetree.c/4820
  73 2983 0    DataIndexTable     21.8182   27015.7099  subfind_density()/subfind_density.c/87
  73 2984 0      DataNodeList     58.1818   27073.8917  subfind_density()/subfind_density.c/89
  73 2985 0              Left    126.8656   27200.7573  subfind_density()/subfind_density.c/91
  73 2986 0             Right    126.8656   27327.6230  subfind_density()/subfind_density.c/92
  73 2987 0              Todo     15.8582   27343.4812  subfind_density()/subfind_density.c/93
  54 2980 0            Father    221.9056   26805.5507  force_treeallocate()/forcetree.c/4827
  54 2981 0           Ngblist     63.4186   26868.9693  subfind_density()/subfind_density.c/78
  54 2982 0         Dist2list    126.8372   26995.8065  subfind_density()/subfind_density.c/79
  73 2988 0             DM_Vx    126.8656   27470.3468  subfind_density()/subfind_density.c/95
  73 2989 0             DM_Vy    126.8656   27597.2125  subfind_density()/subfind_density.c/96
  73 2990 0             DM_Vz    126.8656   27724.0781  subfind_density()/subfind_density.c/97
  54 2983 0    DataIndexTable     21.8182   27017.6247  subfind_density()/subfind_density.c/87
  54 2984 0      DataNodeList     58.1818   27075.8065  subfind_density()/subfind_density.c/89
  54 2985 0              Left    126.8372   27202.6436  subfind_density()/subfind_density.c/91
  73 2991 0    	  DensDataGet    152.7759   27876.8540  subfind_density()/subfind_density.c/161
  73 2992 0  	  DensDataResult    114.5820   27991.4360  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=73: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=8.564 MB).
  54 2986 0             Right    126.8372   27329.4808  subfind_density()/subfind_density.c/92
  54 2987 0              Todo     15.8547   27345.3355  subfind_density()/subfind_density.c/93
  54 2988 0             DM_Vx    126.8372   27472.1726  subfind_density()/subfind_density.c/95
  54 2989 0             DM_Vy    126.8372   27599.0098  subfind_density()/subfind_density.c/96
task 73: endrun called with an error level of 812


  54 2990 0             DM_Vz    126.8372   27725.8470  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 73
  54 2991 0    	  DensDataGet    116.6375   27842.4844  subfind_density()/subfind_density.c/161
  54 2992 0  	  DensDataResult     87.4781   27929.9625  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=54: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.0375 MB).
task 54: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 54
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  50 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  50 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  50 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  50 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  50 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  50 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  50 2973 0         FOF_PList    380.3225   18861.1187  fof_fof()/fof.c/249
  50 2974 0         FOF_GList    633.8708   19494.9896  fof_fof()/fof.c/337
  50 2975 0             Group     12.6628   19507.6524  fof_fof()/fof.c/380
  50 2976 0   DomainNodeIndex      0.3458   19507.9982  force_treeallocate()/forcetree.c/4800
  50 2977 0        Nodes_base   3737.4974   23245.4956  force_treeallocate()/forcetree.c/4803
  50 2978 0     Extnodes_base   3114.5811   26360.0767  force_treeallocate()/forcetree.c/4811
  50 2979 0          Nextnode    222.3007   26582.3774  force_treeallocate()/forcetree.c/4820
  50 2980 0            Father    221.9056   26804.2830  force_treeallocate()/forcetree.c/4827
  50 2981 0           Ngblist     63.3871   26867.6701  subfind_density()/subfind_density.c/78
  50 2982 0         Dist2list    126.7742   26994.4443  subfind_density()/subfind_density.c/79
  50 2983 0    DataIndexTable     21.8182   27016.2625  subfind_density()/subfind_density.c/87
  50 2984 0      DataNodeList     58.1818   27074.4443  subfind_density()/subfind_density.c/89
  50 2985 0              Left    126.7742   27201.2184  subfind_density()/subfind_density.c/91
  50 2986 0             Right    126.7742   27327.9926  subfind_density()/subfind_density.c/92
  50 2987 0              Todo     15.8468   27343.8394  subfind_density()/subfind_density.c/93
  50 2988 0             DM_Vx    126.7742   27470.6135  subfind_density()/subfind_density.c/95
  50 2989 0             DM_Vy    126.7742   27597.3877  subfind_density()/subfind_density.c/96
  50 2990 0             DM_Vz    126.7742   27724.1619  subfind_density()/subfind_density.c/97
  50 2991 0    	  DensDataGet    109.2591   27833.4210  subfind_density()/subfind_density.c/161
  50 2992 0  	  DensDataResult     81.9443   27915.3653  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=50: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.6347 MB).
task 50: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 50
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  78 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  78 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  78 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  78 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  78 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  78 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  78 2973 0         FOF_PList    380.1611   18860.9573  fof_fof()/fof.c/249
  78 2974 0         FOF_GList    633.6018   19494.5591  fof_fof()/fof.c/337
  49 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  49 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  49 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  49 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  49 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  49 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  49 2973 0         FOF_PList    380.2621   18861.0583  fof_fof()/fof.c/249
  49 2974 0         FOF_GList    633.7702   19494.8285  fof_fof()/fof.c/337
  49 2975 0             Group     10.7879   19505.6164  fof_fof()/fof.c/380
  49 2976 0   DomainNodeIndex      0.3458   19505.9622  force_treeallocate()/forcetree.c/4800
  49 2977 0        Nodes_base   3737.4974   23243.4596  force_treeallocate()/forcetree.c/4803
  49 2978 0     Extnodes_base   3114.5811   26358.0407  force_treeallocate()/forcetree.c/4811
  49 2979 0          Nextnode    222.3007   26580.3414  force_treeallocate()/forcetree.c/4820
  49 2980 0            Father    221.9056   26802.2470  force_treeallocate()/forcetree.c/4827
  49 2981 0           Ngblist     63.3770   26865.6240  subfind_density()/subfind_density.c/78
  78 2975 0             Group     11.6906   19506.2497  fof_fof()/fof.c/380
  78 2976 0   DomainNodeIndex      0.3458   19506.5955  force_treeallocate()/forcetree.c/4800
  78 2977 0        Nodes_base   3737.4974   23244.0929  force_treeallocate()/forcetree.c/4803
  49 2982 0         Dist2list    126.7540   26992.3781  subfind_density()/subfind_density.c/79
  49 2983 0    DataIndexTable     21.8182   27014.1963  subfind_density()/subfind_density.c/87
  49 2984 0      DataNodeList     58.1818   27072.3781  subfind_density()/subfind_density.c/89
  49 2985 0              Left    126.7540   27199.1321  subfind_density()/subfind_density.c/91
  49 2986 0             Right    126.7540   27325.8861  subfind_density()/subfind_density.c/92
  78 2978 0     Extnodes_base   3114.5811   26358.6740  force_treeallocate()/forcetree.c/4811
  78 2979 0          Nextnode    222.3007   26580.9747  force_treeallocate()/forcetree.c/4820
  78 2980 0            Father    221.9056   26802.8803  force_treeallocate()/forcetree.c/4827
  49 2987 0              Todo     15.8443   27341.7304  subfind_density()/subfind_density.c/93
  49 2988 0             DM_Vx    126.7540   27468.4844  subfind_density()/subfind_density.c/95
  49 2989 0             DM_Vy    126.7540   27595.2385  subfind_density()/subfind_density.c/96
  49 2990 0             DM_Vz    126.7540   27721.9925  subfind_density()/subfind_density.c/97
  78 2981 0           Ngblist     63.3602   26866.2405  subfind_density()/subfind_density.c/78
  78 2982 0         Dist2list    126.7204   26992.9609  subfind_density()/subfind_density.c/79
  78 2983 0    DataIndexTable     21.8182   27014.7790  subfind_density()/subfind_density.c/87
  78 2984 0      DataNodeList     58.1818   27072.9608  subfind_density()/subfind_density.c/89
  49 2991 0    	  DensDataGet    128.9420   27850.9344  subfind_density()/subfind_density.c/161
  49 2992 0  	  DensDataResult     96.7065   27947.6409  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  78 2985 0              Left    126.7204   27199.6812  subfind_density()/subfind_density.c/91
  78 2986 0             Right    126.7204   27326.4016  subfind_density()/subfind_density.c/92

Task=49: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.3591 MB).
task 49: endrun called with an error level of 812


  78 2987 0              Todo     15.8400   27342.2416  subfind_density()/subfind_density.c/93
  78 2988 0             DM_Vx    126.7204   27468.9620  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 49
  78 2989 0             DM_Vy    126.7204   27595.6823  subfind_density()/subfind_density.c/96
  78 2990 0             DM_Vz    126.7204   27722.4027  subfind_density()/subfind_density.c/97
  78 2991 0    	  DensDataGet    150.1315   27872.5342  subfind_density()/subfind_density.c/161
  78 2992 0  	  DensDataResult    112.5986   27985.1329  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=78: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=14.8671 MB).
task 78: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  53 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  53 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  53 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  53 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  53 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  53 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  53 2973 0         FOF_PList    380.2725   18861.0688  fof_fof()/fof.c/249
  53 2974 0         FOF_GList    633.7876   19494.8564  fof_fof()/fof.c/337
  53 2975 0             Group     12.8589   19507.7153  fof_fof()/fof.c/380
  53 2976 0   DomainNodeIndex      0.3458   19508.0611  force_treeallocate()/forcetree.c/4800
  53 2977 0        Nodes_base   3737.4974   23245.5584  force_treeallocate()/forcetree.c/4803
  53 2978 0     Extnodes_base   3114.5811   26360.1396  force_treeallocate()/forcetree.c/4811
  53 2979 0          Nextnode    222.3007   26582.4403  force_treeallocate()/forcetree.c/4820
  53 2980 0            Father    221.9056   26804.3459  force_treeallocate()/forcetree.c/4827
  53 2981 0           Ngblist     63.3788   26867.7247  subfind_density()/subfind_density.c/78
  53 2982 0         Dist2list    126.7575   26994.4822  subfind_density()/subfind_density.c/79
  74 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  74 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  74 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  74 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  74 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  74 2973 0         FOF_PList    379.9204   18860.7167  fof_fof()/fof.c/249
  74 2974 0         FOF_GList    633.2007   19493.9174  fof_fof()/fof.c/337
  74 2975 0             Group     13.2034   19507.1208  fof_fof()/fof.c/380
  74 2976 0   DomainNodeIndex      0.3458   19507.4666  force_treeallocate()/forcetree.c/4800
  53 2983 0    DataIndexTable     21.8182   27016.3004  subfind_density()/subfind_density.c/87
  53 2984 0      DataNodeList     58.1818   27074.4821  subfind_density()/subfind_density.c/89
  53 2985 0              Left    126.7575   27201.2397  subfind_density()/subfind_density.c/91
  53 2986 0             Right    126.7575   27327.9972  subfind_density()/subfind_density.c/92
  53 2987 0              Todo     15.8447   27343.8419  subfind_density()/subfind_density.c/93
  74 2977 0        Nodes_base   3737.4974   23244.9640  force_treeallocate()/forcetree.c/4803
  74 2978 0     Extnodes_base   3114.5811   26359.5451  force_treeallocate()/forcetree.c/4811
  74 2979 0          Nextnode    222.3007   26581.8458  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  79 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  53 2988 0             DM_Vx    126.7575   27470.5994  subfind_density()/subfind_density.c/95
  53 2989 0             DM_Vy    126.7575   27597.3569  subfind_density()/subfind_density.c/96
  53 2990 0             DM_Vz    126.7575   27724.1144  subfind_density()/subfind_density.c/97
  74 2980 0            Father    221.9056   26803.7514  force_treeallocate()/forcetree.c/4827
  74 2981 0           Ngblist     63.3201   26867.0715  subfind_density()/subfind_density.c/78
  74 2982 0         Dist2list    126.6401   26993.7116  subfind_density()/subfind_density.c/79
  79 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  79 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  79 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  79 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  79 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  79 2973 0         FOF_PList    380.3896   18861.1858  fof_fof()/fof.c/249
  79 2974 0         FOF_GList    633.9826   19495.1684  fof_fof()/fof.c/337
  79 2975 0             Group     12.5062   19507.6746  fof_fof()/fof.c/380
  79 2976 0   DomainNodeIndex      0.3458   19508.0203  force_treeallocate()/forcetree.c/4800
  79 2977 0        Nodes_base   3737.4974   23245.5177  force_treeallocate()/forcetree.c/4803
  79 2978 0     Extnodes_base   3114.5811   26360.0989  force_treeallocate()/forcetree.c/4811
  79 2979 0          Nextnode    222.3007   26582.3996  force_treeallocate()/forcetree.c/4820
  79 2980 0            Father    221.9056   26804.3052  force_treeallocate()/forcetree.c/4827
  53 2991 0    	  DensDataGet    114.8380   27838.9524  subfind_density()/subfind_density.c/161
  53 2992 0  	  DensDataResult     86.1285   27925.0809  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  74 2983 0    DataIndexTable     21.8182   27015.5298  subfind_density()/subfind_density.c/87
  74 2984 0      DataNodeList     58.1818   27073.7116  subfind_density()/subfind_density.c/89
  79 2981 0           Ngblist     63.3983   26867.7035  subfind_density()/subfind_density.c/78
  79 2982 0         Dist2list    126.7965   26994.5000  subfind_density()/subfind_density.c/79
  79 2983 0    DataIndexTable     21.8182   27016.3182  subfind_density()/subfind_density.c/87
  79 2984 0      DataNodeList     58.1818   27074.4999  subfind_density()/subfind_density.c/89

Task=53: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.9191 MB).
task 53: endrun called with an error level of 812


  74 2985 0              Left    126.6401   27200.3518  subfind_density()/subfind_density.c/91
  74 2986 0             Right    126.6401   27326.9919  subfind_density()/subfind_density.c/92
  74 2987 0              Todo     15.8300   27342.8219  subfind_density()/subfind_density.c/93
  79 2985 0              Left    126.7965   27201.2965  subfind_density()/subfind_density.c/91
  79 2986 0             Right    126.7965   27328.0930  subfind_density()/subfind_density.c/92
  79 2987 0              Todo     15.8496   27343.9426  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 53
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  74 2988 0             DM_Vx    126.6401   27469.4621  subfind_density()/subfind_density.c/95
  79 2988 0             DM_Vx    126.7965   27470.7391  subfind_density()/subfind_density.c/95
  79 2989 0             DM_Vy    126.7965   27597.5356  subfind_density()/subfind_density.c/96
  79 2990 0             DM_Vz    126.7965   27724.3321  subfind_density()/subfind_density.c/97
  66 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  66 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  66 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  66 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  66 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  66 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  66 2973 0         FOF_PList    380.2960   18861.0922  fof_fof()/fof.c/249
  66 2974 0         FOF_GList    633.8266   19494.9189  fof_fof()/fof.c/337
  66 2975 0             Group     11.6481   19506.5669  fof_fof()/fof.c/380
  66 2976 0   DomainNodeIndex      0.3458   19506.9127  force_treeallocate()/forcetree.c/4800
  57 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  57 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  57 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  57 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  57 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  57 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  57 2973 0         FOF_PList    380.3779   18861.1741  fof_fof()/fof.c/249
  57 2974 0         FOF_GList    633.9632   19495.1373  fof_fof()/fof.c/337
  57 2975 0             Group     12.9907   19508.1280  fof_fof()/fof.c/380
  57 2976 0   DomainNodeIndex      0.3458   19508.4738  force_treeallocate()/forcetree.c/4800
  57 2977 0        Nodes_base   3737.4974   23245.9712  force_treeallocate()/forcetree.c/4803
  57 2978 0     Extnodes_base   3114.5811   26360.5523  force_treeallocate()/forcetree.c/4811
  57 2979 0          Nextnode    222.3007   26582.8530  force_treeallocate()/forcetree.c/4820
  57 2980 0            Father    221.9056   26804.7586  force_treeallocate()/forcetree.c/4827
  57 2981 0           Ngblist     63.3963   26868.1549  subfind_density()/subfind_density.c/78
  74 2989 0             DM_Vy    126.6401   27596.1022  subfind_density()/subfind_density.c/96
  79 2991 0    	  DensDataGet    130.8174   27855.1495  subfind_density()/subfind_density.c/161
  79 2992 0  	  DensDataResult     98.1130   27953.2626  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=79: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.7374 MB).
  66 2977 0        Nodes_base   3737.4974   23244.4101  force_treeallocate()/forcetree.c/4803
  57 2982 0         Dist2list    126.7926   26994.9476  subfind_density()/subfind_density.c/79
  57 2983 0    DataIndexTable     21.8182   27016.7657  subfind_density()/subfind_density.c/87
  57 2984 0      DataNodeList     58.1818   27074.9475  subfind_density()/subfind_density.c/89
  57 2985 0              Left    126.7926   27201.7402  subfind_density()/subfind_density.c/91
  74 2990 0             DM_Vz    126.6401   27722.7424  subfind_density()/subfind_density.c/97
task 79: endrun called with an error level of 812


  66 2978 0     Extnodes_base   3114.5811   26358.9912  force_treeallocate()/forcetree.c/4811
  66 2979 0          Nextnode    222.3007   26581.2919  force_treeallocate()/forcetree.c/4820
  57 2986 0             Right    126.7926   27328.5328  subfind_density()/subfind_density.c/92
  57 2987 0              Todo     15.8491   27344.3819  subfind_density()/subfind_density.c/93
  57 2988 0             DM_Vx    126.7926   27471.1745  subfind_density()/subfind_density.c/95
  57 2989 0             DM_Vy    126.7926   27597.9672  subfind_density()/subfind_density.c/96
  74 2991 0    	  DensDataGet    109.0305   27831.7728  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 79
  66 2980 0            Father    221.9056   26803.1975  force_treeallocate()/forcetree.c/4827
  66 2981 0           Ngblist     63.3827   26866.5802  subfind_density()/subfind_density.c/78
  66 2982 0         Dist2list    126.7653   26993.3455  subfind_density()/subfind_density.c/79
  57 2990 0             DM_Vz    126.7926   27724.7598  subfind_density()/subfind_density.c/97
  57 2991 0    	  DensDataGet    112.7068   27837.4666  subfind_density()/subfind_density.c/161
  57 2992 0  	  DensDataResult     84.5301   27921.9967  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  74 2992 0  	  DensDataResult     81.7728   27913.5457  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  66 2983 0    DataIndexTable     21.8182   27015.1637  subfind_density()/subfind_density.c/87
  66 2984 0      DataNodeList     58.1818   27073.3455  subfind_density()/subfind_density.c/89

Task=57: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.0033 MB).
task 57: endrun called with an error level of 812



Task=74: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.4543 MB).
task 74: endrun called with an error level of 812


  66 2985 0              Left    126.7653   27200.1108  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 57
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 74
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  66 2986 0             Right    126.7653   27326.8761  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  98 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  98 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  98 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  98 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  98 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  98 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  98 2973 0         FOF_PList    380.4283   18861.2246  fof_fof()/fof.c/249
  66 2987 0              Todo     15.8457   27342.7218  subfind_density()/subfind_density.c/93
  29 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  29 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  29 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  29 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  29 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  29 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  29 2973 0         FOF_PList    380.0170   18860.8132  fof_fof()/fof.c/249
  29 2974 0         FOF_GList    633.3616   19494.1748  fof_fof()/fof.c/337
  29 2975 0             Group     11.3752   19505.5500  fof_fof()/fof.c/380
  29 2976 0   DomainNodeIndex      0.3458   19505.8957  force_treeallocate()/forcetree.c/4800
  29 2977 0        Nodes_base   3737.4974   23243.3931  force_treeallocate()/forcetree.c/4803
  29 2978 0     Extnodes_base   3114.5811   26357.9742  force_treeallocate()/forcetree.c/4811
  29 2979 0          Nextnode    222.3007   26580.2750  force_treeallocate()/forcetree.c/4820
  98 2974 0         FOF_GList    634.0472   19495.2718  fof_fof()/fof.c/337
  98 2975 0             Group     12.6514   19507.9232  fof_fof()/fof.c/380
  98 2976 0   DomainNodeIndex      0.3458   19508.2690  force_treeallocate()/forcetree.c/4800
  66 2988 0             DM_Vx    126.7653   27469.4871  subfind_density()/subfind_density.c/95
  29 2980 0            Father    221.9056   26802.1806  force_treeallocate()/forcetree.c/4827
  29 2981 0           Ngblist     63.3362   26865.5167  subfind_density()/subfind_density.c/78
  29 2982 0         Dist2list    126.6723   26992.1890  subfind_density()/subfind_density.c/79
  98 2977 0        Nodes_base   3737.4974   23245.7664  force_treeallocate()/forcetree.c/4803
  98 2978 0     Extnodes_base   3114.5811   26360.3475  force_treeallocate()/forcetree.c/4811
  98 2979 0          Nextnode    222.3007   26582.6483  force_treeallocate()/forcetree.c/4820
  66 2989 0             DM_Vy    126.7653   27596.2525  subfind_density()/subfind_density.c/96
  29 2983 0    DataIndexTable     21.8182   27014.0072  subfind_density()/subfind_density.c/87
  29 2984 0      DataNodeList     58.1818   27072.1890  subfind_density()/subfind_density.c/89
  29 2985 0              Left    126.6723   27198.8613  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  98 2980 0            Father    221.9056   26804.5538  force_treeallocate()/forcetree.c/4827
  98 2981 0           Ngblist     63.4047   26867.9586  subfind_density()/subfind_density.c/78
  98 2982 0         Dist2list    126.8094   26994.7680  subfind_density()/subfind_density.c/79
  66 2990 0             DM_Vz    126.7653   27723.0178  subfind_density()/subfind_density.c/97
  29 2986 0             Right    126.6723   27325.5336  subfind_density()/subfind_density.c/92
  29 2987 0              Todo     15.8340   27341.3677  subfind_density()/subfind_density.c/93
  60 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  60 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  60 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  60 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  60 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  60 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  60 2973 0         FOF_PList    380.6859   18861.4821  fof_fof()/fof.c/249
  60 2974 0         FOF_GList    634.4765   19495.9586  fof_fof()/fof.c/337
  60 2975 0             Group     11.4229   19507.3815  fof_fof()/fof.c/380
  60 2976 0   DomainNodeIndex      0.3458   19507.7273  force_treeallocate()/forcetree.c/4800
  98 2983 0    DataIndexTable     21.8182   27016.5862  subfind_density()/subfind_density.c/87
  98 2984 0      DataNodeList     58.1818   27074.7680  subfind_density()/subfind_density.c/89
  98 2985 0              Left    126.8094   27201.5774  subfind_density()/subfind_density.c/91
  66 2991 0    	  DensDataGet    117.7937   27840.8115  subfind_density()/subfind_density.c/161
  66 2992 0  	  DensDataResult     88.3453   27929.1568  subfind_density()/subfind_density.c/201
  29 2988 0             DM_Vx    126.6723   27468.0400  subfind_density()/subfind_density.c/95
  29 2989 0             DM_Vy    126.6723   27594.7123  subfind_density()/subfind_density.c/96
  29 2990 0             DM_Vz    126.6723   27721.3846  subfind_density()/subfind_density.c/97
  60 2977 0        Nodes_base   3737.4974   23245.2246  force_treeallocate()/forcetree.c/4803
  60 2978 0     Extnodes_base   3114.5811   26359.8058  force_treeallocate()/forcetree.c/4811
  98 2986 0             Right    126.8094   27328.3869  subfind_density()/subfind_density.c/92
----------------------------------------------------------------------------------------
  29 2991 0    	  DensDataGet    115.6804   27837.0650  subfind_density()/subfind_density.c/161
  60 2979 0          Nextnode    222.3007   26582.1065  force_treeallocate()/forcetree.c/4820
  60 2980 0            Father    221.9056   26804.0121  force_treeallocate()/forcetree.c/4827
  60 2981 0           Ngblist     63.4476   26867.4597  subfind_density()/subfind_density.c/78
  60 2982 0         Dist2list    126.8953   26994.3550  subfind_density()/subfind_density.c/79
  98 2987 0              Todo     15.8512   27344.2381  subfind_density()/subfind_density.c/93

Task=66: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.8432 MB).
  29 2992 0  	  DensDataResult     86.7603   27923.8253  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  60 2983 0    DataIndexTable     21.8182   27016.1732  subfind_density()/subfind_density.c/87
  60 2984 0      DataNodeList     58.1818   27074.3550  subfind_density()/subfind_density.c/89
  60 2985 0              Left    126.8953   27201.2503  subfind_density()/subfind_density.c/91
  98 2988 0             DM_Vx    126.8094   27471.0475  subfind_density()/subfind_density.c/95
task 66: endrun called with an error level of 812



Task=29: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.1747 MB).
task 29: endrun called with an error level of 812


  60 2986 0             Right    126.8953   27328.1456  subfind_density()/subfind_density.c/92
  60 2987 0              Todo     15.8619   27344.0075  subfind_density()/subfind_density.c/93
  60 2988 0             DM_Vx    126.8953   27470.9028  subfind_density()/subfind_density.c/95
  98 2989 0             DM_Vy    126.8094   27597.8570  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 66
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 29
  60 2989 0             DM_Vy    126.8953   27597.7981  subfind_density()/subfind_density.c/96
  98 2990 0             DM_Vz    126.8094   27724.6664  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  60 2990 0             DM_Vz    126.8953   27724.6934  subfind_density()/subfind_density.c/97
  98 2991 0    	  DensDataGet    122.6398   27847.3062  subfind_density()/subfind_density.c/161
  67 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  67 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  67 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  67 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  67 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  67 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  67 2973 0         FOF_PList    380.2684   18861.0646  fof_fof()/fof.c/249
  67 2974 0         FOF_GList    633.7806   19494.8452  fof_fof()/fof.c/337
  67 2975 0             Group     11.7871   19506.6323  fof_fof()/fof.c/380
  67 2976 0   DomainNodeIndex      0.3458   19506.9781  force_treeallocate()/forcetree.c/4800
  67 2977 0        Nodes_base   3737.4974   23244.4754  force_treeallocate()/forcetree.c/4803
  67 2978 0     Extnodes_base   3114.5811   26359.0566  force_treeallocate()/forcetree.c/4811
  60 2991 0    	  DensDataGet    108.8657   27833.5591  subfind_density()/subfind_density.c/161
  98 2992 0  	  DensDataResult     91.9798   27939.2860  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  67 2979 0          Nextnode    222.3007   26581.3573  force_treeallocate()/forcetree.c/4820
  67 2980 0            Father    221.9056   26803.2629  force_treeallocate()/forcetree.c/4827
  67 2981 0           Ngblist     63.3781   26866.6410  subfind_density()/subfind_density.c/78
  60 2992 0  	  DensDataResult     81.6492   27915.2083  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=60: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.7917 MB).

Task=98: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.714 MB).
  67 2982 0         Dist2list    126.7561   26993.3971  subfind_density()/subfind_density.c/79
  67 2983 0    DataIndexTable     21.8182   27015.2152  subfind_density()/subfind_density.c/87
  67 2984 0      DataNodeList     58.1818   27073.3970  subfind_density()/subfind_density.c/89
  67 2985 0              Left    126.7561   27200.1532  subfind_density()/subfind_density.c/91
task 60: endrun called with an error level of 812


task 98: endrun called with an error level of 812


  67 2986 0             Right    126.7561   27326.9093  subfind_density()/subfind_density.c/92
  67 2987 0              Todo     15.8445   27342.7538  subfind_density()/subfind_density.c/93
  67 2988 0             DM_Vx    126.7561   27469.5099  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 60
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 98
  67 2989 0             DM_Vy    126.7561   27596.2660  subfind_density()/subfind_density.c/96
  67 2990 0             DM_Vz    126.7561   27723.0222  subfind_density()/subfind_density.c/97
  67 2991 0    	  DensDataGet    115.7064   27838.7285  subfind_density()/subfind_density.c/161
  67 2992 0  	  DensDataResult     86.7798   27925.5083  subfind_density()/subfind_density.c/201
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
  56 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  56 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  56 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  56 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  56 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  56 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570

Task=67: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.4917 MB).
  56 2973 0         FOF_PList    380.5828   18861.3790  fof_fof()/fof.c/249
  56 2974 0         FOF_GList    634.3046   19495.6836  fof_fof()/fof.c/337
  56 2975 0             Group     12.3557   19508.0393  fof_fof()/fof.c/380
task 67: endrun called with an error level of 812


  56 2976 0   DomainNodeIndex      0.3458   19508.3851  force_treeallocate()/forcetree.c/4800
  56 2977 0        Nodes_base   3737.4974   23245.8825  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 67
  56 2978 0     Extnodes_base   3114.5811   26360.4636  force_treeallocate()/forcetree.c/4811
  56 2979 0          Nextnode    222.3007   26582.7644  force_treeallocate()/forcetree.c/4820
  56 2980 0            Father    221.9056   26804.6699  force_treeallocate()/forcetree.c/4827
  56 2981 0           Ngblist     63.4305   26868.1004  subfind_density()/subfind_density.c/78
  56 2982 0         Dist2list    126.8609   26994.9613  subfind_density()/subfind_density.c/79
  56 2983 0    DataIndexTable     21.8182   27016.7795  subfind_density()/subfind_density.c/87
  56 2984 0      DataNodeList     58.1818   27074.9613  subfind_density()/subfind_density.c/89
  56 2985 0              Left    126.8609   27201.8222  subfind_density()/subfind_density.c/91
  56 2986 0             Right    126.8609   27328.6831  subfind_density()/subfind_density.c/92
  56 2987 0              Todo     15.8576   27344.5408  subfind_density()/subfind_density.c/93
  56 2988 0             DM_Vx    126.8609   27471.4017  subfind_density()/subfind_density.c/95
  56 2989 0             DM_Vy    126.8609   27598.2626  subfind_density()/subfind_density.c/96
  56 2990 0             DM_Vz    126.8609   27725.1235  subfind_density()/subfind_density.c/97
  56 2991 0    	  DensDataGet    129.2573   27854.3809  subfind_density()/subfind_density.c/161
  56 2992 0  	  DensDataResult     96.9430   27951.3239  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=56: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=48.6761 MB).
task 56: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 56
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 56:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 56: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 56
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  65 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  65 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  65 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  65 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  65 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  65 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  65 2973 0         FOF_PList    379.8723   18860.6685  fof_fof()/fof.c/249
  65 2974 0         FOF_GList    633.1205   19493.7890  fof_fof()/fof.c/337
  65 2975 0             Group     11.1666   19504.9557  fof_fof()/fof.c/380
  65 2976 0   DomainNodeIndex      0.3458   19505.3014  force_treeallocate()/forcetree.c/4800
  65 2977 0        Nodes_base   3737.4974   23242.7988  force_treeallocate()/forcetree.c/4803
  65 2978 0     Extnodes_base   3114.5811   26357.3799  force_treeallocate()/forcetree.c/4811
  65 2979 0          Nextnode    222.3007   26579.6807  force_treeallocate()/forcetree.c/4820
  65 2980 0            Father    221.9056   26801.5863  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  62 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  65 2981 0           Ngblist     63.3120   26864.8983  subfind_density()/subfind_density.c/78
  65 2982 0         Dist2list    126.6241   26991.5224  subfind_density()/subfind_density.c/79
  65 2983 0    DataIndexTable     21.8182   27013.3406  subfind_density()/subfind_density.c/87
  65 2984 0      DataNodeList     58.1818   27071.5224  subfind_density()/subfind_density.c/89
  65 2985 0              Left    126.6241   27198.1465  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  62 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  62 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  62 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  62 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  62 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  62 2973 0         FOF_PList    380.6476   18861.4439  fof_fof()/fof.c/249
  62 2974 0         FOF_GList    634.4127   19495.8566  fof_fof()/fof.c/337
  62 2975 0             Group     10.5897   19506.4463  fof_fof()/fof.c/380
  62 2976 0   DomainNodeIndex      0.3458   19506.7921  force_treeallocate()/forcetree.c/4800
  62 2977 0        Nodes_base   3737.4974   23244.2895  force_treeallocate()/forcetree.c/4803
  62 2978 0     Extnodes_base   3114.5811   26358.8706  force_treeallocate()/forcetree.c/4811
  62 2979 0          Nextnode    222.3007   26581.1713  force_treeallocate()/forcetree.c/4820
  65 2986 0             Right    126.6241   27324.7706  subfind_density()/subfind_density.c/92
  65 2987 0              Todo     15.8280   27340.5986  subfind_density()/subfind_density.c/93
  65 2988 0             DM_Vx    126.6241   27467.2227  subfind_density()/subfind_density.c/95
  65 2989 0             DM_Vy    126.6241   27593.8468  subfind_density()/subfind_density.c/96
  51 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  51 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  51 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  51 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  51 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  51 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  51 2973 0         FOF_PList    380.1924   18860.9886  fof_fof()/fof.c/249
  51 2974 0         FOF_GList    633.6539   19494.6425  fof_fof()/fof.c/337
  51 2975 0             Group     10.9311   19505.5736  fof_fof()/fof.c/380
  51 2976 0   DomainNodeIndex      0.3458   19505.9194  force_treeallocate()/forcetree.c/4800
  51 2977 0        Nodes_base   3737.4974   23243.4168  force_treeallocate()/forcetree.c/4803
  51 2978 0     Extnodes_base   3114.5811   26357.9979  force_treeallocate()/forcetree.c/4811
  51 2979 0          Nextnode    222.3007   26580.2987  force_treeallocate()/forcetree.c/4820
  51 2980 0            Father    221.9056   26802.2042  force_treeallocate()/forcetree.c/4827
  51 2981 0           Ngblist     63.3654   26865.5696  subfind_density()/subfind_density.c/78
  51 2982 0         Dist2list    126.7308   26992.3004  subfind_density()/subfind_density.c/79
  76 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  76 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  76 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  76 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  76 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  76 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  76 2973 0         FOF_PList    380.6414   18861.4376  fof_fof()/fof.c/249
  76 2974 0         FOF_GList    634.4023   19495.8399  fof_fof()/fof.c/337
  76 2975 0             Group     11.3970   19507.2368  fof_fof()/fof.c/380
  62 2980 0            Father    221.9056   26803.0769  force_treeallocate()/forcetree.c/4827
  62 2981 0           Ngblist     63.4413   26866.5182  subfind_density()/subfind_density.c/78
  62 2982 0         Dist2list    126.8825   26993.4007  subfind_density()/subfind_density.c/79
  65 2990 0             DM_Vz    126.6241   27720.4709  subfind_density()/subfind_density.c/97
  65 2991 0    	  DensDataGet    117.7285   27838.1994  subfind_density()/subfind_density.c/161
  65 2992 0  	  DensDataResult     88.2964   27926.4958  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  51 2983 0    DataIndexTable     21.8182   27014.1186  subfind_density()/subfind_density.c/87
  51 2984 0      DataNodeList     58.1818   27072.3004  subfind_density()/subfind_density.c/89
  51 2985 0              Left    126.7308   27199.0312  subfind_density()/subfind_density.c/91
  51 2986 0             Right    126.7308   27325.7620  subfind_density()/subfind_density.c/92
  76 2976 0   DomainNodeIndex      0.3458   19507.5826  force_treeallocate()/forcetree.c/4800
  76 2977 0        Nodes_base   3737.4974   23245.0800  force_treeallocate()/forcetree.c/4803
  76 2978 0     Extnodes_base   3114.5811   26359.6611  force_treeallocate()/forcetree.c/4811
  62 2983 0    DataIndexTable     21.8182   27015.2189  subfind_density()/subfind_density.c/87
  62 2984 0      DataNodeList     58.1818   27073.4007  subfind_density()/subfind_density.c/89
  62 2985 0              Left    126.8825   27200.2833  subfind_density()/subfind_density.c/91

Task=65: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.5042 MB).
task 65: endrun called with an error level of 812


  51 2987 0              Todo     15.8414   27341.6033  subfind_density()/subfind_density.c/93
  51 2988 0             DM_Vx    126.7308   27468.3341  subfind_density()/subfind_density.c/95
  51 2989 0             DM_Vy    126.7308   27595.0649  subfind_density()/subfind_density.c/96
  76 2979 0          Nextnode    222.3007   26581.9619  force_treeallocate()/forcetree.c/4820
  76 2980 0            Father    221.9056   26803.8674  force_treeallocate()/forcetree.c/4827
  62 2986 0             Right    126.8825   27327.1658  subfind_density()/subfind_density.c/92
  62 2987 0              Todo     15.8603   27343.0261  subfind_density()/subfind_density.c/93
  62 2988 0             DM_Vx    126.8825   27469.9087  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 65
  51 2990 0             DM_Vz    126.7308   27721.7957  subfind_density()/subfind_density.c/97
  51 2991 0    	  DensDataGet    120.8875   27842.6832  subfind_density()/subfind_density.c/161
  51 2992 0  	  DensDataResult     90.6656   27933.3488  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  76 2981 0           Ngblist     63.4402   26867.3077  subfind_density()/subfind_density.c/78
  76 2982 0         Dist2list    126.8805   26994.1881  subfind_density()/subfind_density.c/79
  62 2989 0             DM_Vy    126.8825   27596.7912  subfind_density()/subfind_density.c/96
  62 2990 0             DM_Vz    126.8825   27723.6738  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=51: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.6512 MB).
task 51: endrun called with an error level of 812


  76 2983 0    DataIndexTable     21.8182   27016.0063  subfind_density()/subfind_density.c/87
  76 2984 0      DataNodeList     58.1818   27074.1881  subfind_density()/subfind_density.c/89
  62 2991 0    	  DensDataGet    125.6854   27849.3591  subfind_density()/subfind_density.c/161
  64 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  64 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  64 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  64 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  64 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  64 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 51
  76 2985 0              Left    126.8805   27201.0686  subfind_density()/subfind_density.c/91
  62 2992 0  	  DensDataResult     94.2640   27943.6231  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  64 2973 0         FOF_PList    380.6853   18861.4815  fof_fof()/fof.c/249
  64 2974 0         FOF_GList    634.4755   19495.9570  fof_fof()/fof.c/337
  64 2975 0             Group     11.7923   19507.7493  fof_fof()/fof.c/380
  64 2976 0   DomainNodeIndex      0.3458   19508.0951  force_treeallocate()/forcetree.c/4800
  76 2986 0             Right    126.8805   27327.9490  subfind_density()/subfind_density.c/92

Task=62: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.3769 MB).
task 62: endrun called with an error level of 812


  64 2977 0        Nodes_base   3737.4974   23245.5924  force_treeallocate()/forcetree.c/4803
  64 2978 0     Extnodes_base   3114.5811   26360.1736  force_treeallocate()/forcetree.c/4811
  64 2979 0          Nextnode    222.3007   26582.4743  force_treeallocate()/forcetree.c/4820
  76 2987 0              Todo     15.8601   27343.8091  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  64 2980 0            Father    221.9056   26804.3799  force_treeallocate()/forcetree.c/4827
  64 2981 0           Ngblist     63.4475   26867.8274  subfind_density()/subfind_density.c/78
  64 2982 0         Dist2list    126.8951   26994.7225  subfind_density()/subfind_density.c/79
  76 2988 0             DM_Vx    126.8805   27470.6895  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 62
  64 2983 0    DataIndexTable     21.8182   27016.5407  subfind_density()/subfind_density.c/87
  64 2984 0      DataNodeList     58.1818   27074.7225  subfind_density()/subfind_density.c/89
  64 2985 0              Left    126.8951   27201.6176  subfind_density()/subfind_density.c/91
  76 2989 0             DM_Vy    126.8805   27597.5700  subfind_density()/subfind_density.c/96
  63 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  63 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  63 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  63 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  63 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  63 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  63 2973 0         FOF_PList    380.0002   18860.7964  fof_fof()/fof.c/249
  63 2974 0         FOF_GList    633.3336   19494.1300  fof_fof()/fof.c/337
  63 2975 0             Group     11.3627   19505.4927  fof_fof()/fof.c/380
  63 2976 0   DomainNodeIndex      0.3458   19505.8385  force_treeallocate()/forcetree.c/4800
  63 2977 0        Nodes_base   3737.4974   23243.3358  force_treeallocate()/forcetree.c/4803
  63 2978 0     Extnodes_base   3114.5811   26357.9170  force_treeallocate()/forcetree.c/4811
  63 2979 0          Nextnode    222.3007   26580.2177  force_treeallocate()/forcetree.c/4820
  63 2980 0            Father    221.9056   26802.1233  force_treeallocate()/forcetree.c/4827
  64 2986 0             Right    126.8951   27328.5127  subfind_density()/subfind_density.c/92
  64 2987 0              Todo     15.8619   27344.3746  subfind_density()/subfind_density.c/93
  64 2988 0             DM_Vx    126.8951   27471.2697  subfind_density()/subfind_density.c/95
  64 2989 0             DM_Vy    126.8951   27598.1648  subfind_density()/subfind_density.c/96
  76 2990 0             DM_Vz    126.8805   27724.4504  subfind_density()/subfind_density.c/97
  63 2981 0           Ngblist     63.3334   26865.4567  subfind_density()/subfind_density.c/78
  63 2982 0         Dist2list    126.6667   26992.1234  subfind_density()/subfind_density.c/79
  64 2990 0             DM_Vz    126.8951   27725.0599  subfind_density()/subfind_density.c/97
  64 2991 0    	  DensDataGet    116.5815   27841.6414  subfind_density()/subfind_density.c/161
  76 2991 0    	  DensDataGet    117.3436   27841.7941  subfind_density()/subfind_density.c/161
  63 2983 0    DataIndexTable     21.8182   27013.9416  subfind_density()/subfind_density.c/87
  63 2984 0      DataNodeList     58.1818   27072.1234  subfind_density()/subfind_density.c/89
  64 2992 0  	  DensDataResult     87.4362   27929.0776  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  76 2992 0  	  DensDataResult     88.0077   27929.8018  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  63 2985 0              Left    126.6667   27198.7901  subfind_density()/subfind_density.c/91

Task=64: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.9224 MB).
task 64: endrun called with an error level of 812



  63 2986 0             Right    126.6667   27325.4568  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 64

Task=76: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.1982 MB).
  63 2987 0              Todo     15.8333   27341.2901  subfind_density()/subfind_density.c/93
 An HDF5 error was detected, we'd better stop.
task 76: endrun called with an error level of 812


  63 2988 0             DM_Vx    126.6667   27467.9568  subfind_density()/subfind_density.c/95
HDF5-DIAG: Error detected in HDF5 (1.8.12) application called MPI_Abort(MPI_COMM_WORLD, 812) - process 76
  63 2989 0             DM_Vy    126.6667   27594.6236  subfind_density()/subfind_density.c/96
MPI-process 64:
  63 2990 0             DM_Vz    126.6667   27721.2903  subfind_density()/subfind_density.c/97
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
  63 2991 0    	  DensDataGet    115.6434   27836.9337  subfind_density()/subfind_density.c/161
task 64: endrun called with an error level of 124


  63 2992 0  	  DensDataResult     86.7325   27923.6662  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 64

Task=63: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.3338 MB).
task 63: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 63
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  69 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  69 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  69 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  69 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  69 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  69 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  69 2973 0         FOF_PList    380.0506   18860.8468  fof_fof()/fof.c/249
  69 2974 0         FOF_GList    633.4177   19494.2645  fof_fof()/fof.c/337
  69 2975 0             Group     11.0390   19505.3035  fof_fof()/fof.c/380
  69 2976 0   DomainNodeIndex      0.3458   19505.6493  force_treeallocate()/forcetree.c/4800
  69 2977 0        Nodes_base   3737.4974   23243.1466  force_treeallocate()/forcetree.c/4803
  69 2978 0     Extnodes_base   3114.5811   26357.7278  force_treeallocate()/forcetree.c/4811
  69 2979 0          Nextnode    222.3007   26580.0285  force_treeallocate()/forcetree.c/4820
  69 2980 0            Father    221.9056   26801.9341  force_treeallocate()/forcetree.c/4827
  69 2981 0           Ngblist     63.3418   26865.2759  subfind_density()/subfind_density.c/78
  59 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  59 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  59 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  59 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  59 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  59 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  59 2973 0         FOF_PList    380.3326   18861.1288  fof_fof()/fof.c/249
  59 2974 0         FOF_GList    633.8876   19495.0164  fof_fof()/fof.c/337
  59 2975 0             Group     12.7780   19507.7945  fof_fof()/fof.c/380
  59 2976 0   DomainNodeIndex      0.3458   19508.1402  force_treeallocate()/forcetree.c/4800
  69 2982 0         Dist2list    126.6835   26991.9594  subfind_density()/subfind_density.c/79
  69 2983 0    DataIndexTable     21.8182   27013.7776  subfind_density()/subfind_density.c/87
  69 2984 0      DataNodeList     58.1818   27071.9594  subfind_density()/subfind_density.c/89
  69 2985 0              Left    126.6835   27198.6429  subfind_density()/subfind_density.c/91
  59 2977 0        Nodes_base   3737.4974   23245.6376  force_treeallocate()/forcetree.c/4803
  59 2978 0     Extnodes_base   3114.5811   26360.2187  force_treeallocate()/forcetree.c/4811
  59 2979 0          Nextnode    222.3007   26582.5195  force_treeallocate()/forcetree.c/4820
  59 2980 0            Father    221.9056   26804.4251  force_treeallocate()/forcetree.c/4827
  69 2986 0             Right    126.6835   27325.3264  subfind_density()/subfind_density.c/92
  69 2987 0              Todo     15.8354   27341.1619  subfind_density()/subfind_density.c/93
  69 2988 0             DM_Vx    126.6835   27467.8454  subfind_density()/subfind_density.c/95
  69 2989 0             DM_Vy    126.6835   27594.5289  subfind_density()/subfind_density.c/96
  59 2981 0           Ngblist     63.3888   26867.8138  subfind_density()/subfind_density.c/78
  59 2982 0         Dist2list    126.7775   26994.5914  subfind_density()/subfind_density.c/79
  59 2983 0    DataIndexTable     21.8182   27016.4095  subfind_density()/subfind_density.c/87
  69 2990 0             DM_Vz    126.6835   27721.2125  subfind_density()/subfind_density.c/97
  69 2991 0    	  DensDataGet    119.3987   27840.6112  subfind_density()/subfind_density.c/161
  69 2992 0  	  DensDataResult     89.5491   27930.1603  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  59 2984 0      DataNodeList     58.1818   27074.5913  subfind_density()/subfind_density.c/89
  59 2985 0              Left    126.7775   27201.3689  subfind_density()/subfind_density.c/91
  59 2986 0             Right    126.7775   27328.1464  subfind_density()/subfind_density.c/92


Task=69: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.8397 MB).
task 69: endrun called with an error level of 812


  59 2987 0              Todo     15.8472   27343.9936  subfind_density()/subfind_density.c/93
  59 2988 0             DM_Vx    126.7775   27470.7711  subfind_density()/subfind_density.c/95
  59 2989 0             DM_Vy    126.7775   27597.5486  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 69
  59 2990 0             DM_Vz    126.7775   27724.3261  subfind_density()/subfind_density.c/97
  59 2991 0    	  DensDataGet    127.9647   27852.2908  subfind_density()/subfind_density.c/161
  59 2992 0  	  DensDataResult     95.9735   27948.2643  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=59: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.7357 MB).
task 59: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 59
  99 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  99 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  99 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  99 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  99 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  99 2973 0         FOF_PList    380.6073   18861.4035  fof_fof()/fof.c/249
  99 2974 0         FOF_GList    634.3455   19495.7491  fof_fof()/fof.c/337
  99 2975 0             Group     13.2397   19508.9888  fof_fof()/fof.c/380
  99 2976 0   DomainNodeIndex      0.3458   19509.3346  force_treeallocate()/forcetree.c/4800
  99 2977 0        Nodes_base   3737.4974   23246.8319  force_treeallocate()/forcetree.c/4803
  99 2978 0     Extnodes_base   3114.5811   26361.4131  force_treeallocate()/forcetree.c/4811
  99 2979 0          Nextnode    222.3007   26583.7138  force_treeallocate()/forcetree.c/4820
  99 2980 0            Father    221.9056   26805.6194  force_treeallocate()/forcetree.c/4827
  99 2981 0           Ngblist     63.4346   26869.0540  subfind_density()/subfind_density.c/78
  99 2982 0         Dist2list    126.8691   26995.9231  subfind_density()/subfind_density.c/79
  99 2983 0    DataIndexTable     21.8182   27017.7412  subfind_density()/subfind_density.c/87
  99 2984 0      DataNodeList     58.1818   27075.9230  subfind_density()/subfind_density.c/89
  99 2985 0              Left    126.8691   27202.7921  subfind_density()/subfind_density.c/91
  99 2986 0             Right    126.8691   27329.6612  subfind_density()/subfind_density.c/92
  99 2987 0              Todo     15.8586   27345.5199  subfind_density()/subfind_density.c/93
  99 2988 0             DM_Vx    126.8691   27472.3890  subfind_density()/subfind_density.c/95
  99 2989 0             DM_Vy    126.8691   27599.2581  subfind_density()/subfind_density.c/96
  99 2990 0             DM_Vz    126.8691   27726.1272  subfind_density()/subfind_density.c/97
  99 2991 0    	  DensDataGet    110.5204   27836.6476  subfind_density()/subfind_density.c/161
  99 2992 0  	  DensDataResult     82.8903   27919.5380  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=99: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.462 MB).
task 99: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 99
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  25 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  25 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  25 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  25 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  25 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  25 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  25 2973 0         FOF_PList    380.5821   18861.3783  fof_fof()/fof.c/249
  25 2974 0         FOF_GList    634.3034   19495.6817  fof_fof()/fof.c/337
  25 2975 0             Group     12.4429   19508.1246  fof_fof()/fof.c/380
  25 2976 0   DomainNodeIndex      0.3458   19508.4704  force_treeallocate()/forcetree.c/4800
  25 2977 0        Nodes_base   3737.4974   23245.9677  force_treeallocate()/forcetree.c/4803
  25 2978 0     Extnodes_base   3114.5811   26360.5489  force_treeallocate()/forcetree.c/4811
 102 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 102 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 102 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 102 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 102 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 102 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 102 2973 0         FOF_PList    380.6035   18861.3998  fof_fof()/fof.c/249
 102 2974 0         FOF_GList    634.3392   19495.7390  fof_fof()/fof.c/337
 102 2975 0             Group     10.7547   19506.4937  fof_fof()/fof.c/380
  25 2979 0          Nextnode    222.3007   26582.8496  force_treeallocate()/forcetree.c/4820
  25 2980 0            Father    221.9056   26804.7552  force_treeallocate()/forcetree.c/4827
  25 2981 0           Ngblist     63.4303   26868.1855  subfind_density()/subfind_density.c/78
 102 2976 0   DomainNodeIndex      0.3458   19506.8394  force_treeallocate()/forcetree.c/4800
 102 2977 0        Nodes_base   3737.4974   23244.3368  force_treeallocate()/forcetree.c/4803
 102 2978 0     Extnodes_base   3114.5811   26358.9180  force_treeallocate()/forcetree.c/4811
  25 2982 0         Dist2list    126.8607   26995.0462  subfind_density()/subfind_density.c/79
  25 2983 0    DataIndexTable     21.8182   27016.8644  subfind_density()/subfind_density.c/87
  25 2984 0      DataNodeList     58.1818   27075.0462  subfind_density()/subfind_density.c/89
 102 2979 0          Nextnode    222.3007   26581.2187  force_treeallocate()/forcetree.c/4820
 102 2980 0            Father    221.9056   26803.1243  force_treeallocate()/forcetree.c/4827
  25 2985 0              Left    126.8607   27201.9069  subfind_density()/subfind_density.c/91
  25 2986 0             Right    126.8607   27328.7676  subfind_density()/subfind_density.c/92
  25 2987 0              Todo     15.8576   27344.6252  subfind_density()/subfind_density.c/93
 102 2981 0           Ngblist     63.4339   26866.5582  subfind_density()/subfind_density.c/78
 102 2982 0         Dist2list    126.8678   26993.4260  subfind_density()/subfind_density.c/79
 102 2983 0    DataIndexTable     21.8182   27015.2442  subfind_density()/subfind_density.c/87
  25 2988 0             DM_Vx    126.8607   27471.4859  subfind_density()/subfind_density.c/95
  25 2989 0             DM_Vy    126.8607   27598.3465  subfind_density()/subfind_density.c/96
 102 2984 0      DataNodeList     58.1818   27073.4260  subfind_density()/subfind_density.c/89
 102 2985 0              Left    126.8678   27200.2939  subfind_density()/subfind_density.c/91
 102 2986 0             Right    126.8678   27327.1617  subfind_density()/subfind_density.c/92
  25 2990 0             DM_Vz    126.8607   27725.2072  subfind_density()/subfind_density.c/97
 102 2987 0              Todo     15.8585   27343.0202  subfind_density()/subfind_density.c/93
  25 2991 0    	  DensDataGet    116.8109   27842.0181  subfind_density()/subfind_density.c/161
 102 2988 0             DM_Vx    126.8678   27469.8880  subfind_density()/subfind_density.c/95
  25 2992 0  	  DensDataResult     87.6082   27929.6263  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 102 2989 0             DM_Vy    126.8678   27596.7559  subfind_density()/subfind_density.c/96

Task=25: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.3737 MB).
task 25: endrun called with an error level of 812


 102 2990 0             DM_Vz    126.8678   27723.6237  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 25
 102 2991 0    	  DensDataGet    128.6357   27852.2595  subfind_density()/subfind_density.c/161
 102 2992 0  	  DensDataResult     96.4768   27948.7363  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=102: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.2637 MB).
task 102: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 102
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  30 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  30 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  30 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  30 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  30 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  30 2973 0         FOF_PList    380.5793   18861.3755  fof_fof()/fof.c/249
  30 2974 0         FOF_GList    634.2989   19495.6744  fof_fof()/fof.c/337
  30 2975 0             Group     12.4844   19508.1588  fof_fof()/fof.c/380
  30 2976 0   DomainNodeIndex      0.3458   19508.5045  force_treeallocate()/forcetree.c/4800
  30 2977 0        Nodes_base   3737.4974   23246.0019  force_treeallocate()/forcetree.c/4803
  30 2978 0     Extnodes_base   3114.5811   26360.5831  force_treeallocate()/forcetree.c/4811
  30 2979 0          Nextnode    222.3007   26582.8838  force_treeallocate()/forcetree.c/4820
  30 2980 0            Father    221.9056   26804.7894  force_treeallocate()/forcetree.c/4827
  30 2981 0           Ngblist     63.4299   26868.2193  subfind_density()/subfind_density.c/78
  30 2982 0         Dist2list    126.8598   26995.0790  subfind_density()/subfind_density.c/79
  30 2983 0    DataIndexTable     21.8182   27016.8972  subfind_density()/subfind_density.c/87
  30 2984 0      DataNodeList     58.1818   27075.0790  subfind_density()/subfind_density.c/89
  30 2985 0              Left    126.8598   27201.9388  subfind_density()/subfind_density.c/91
  30 2986 0             Right    126.8598   27328.7986  subfind_density()/subfind_density.c/92
  30 2987 0              Todo     15.8575   27344.6560  subfind_density()/subfind_density.c/93
  30 2988 0             DM_Vx    126.8598   27471.5158  subfind_density()/subfind_density.c/95
  30 2989 0             DM_Vy    126.8598   27598.3756  subfind_density()/subfind_density.c/96
  30 2990 0             DM_Vz    126.8598   27725.2353  subfind_density()/subfind_density.c/97
  30 2991 0    	  DensDataGet    128.5660   27853.8013  subfind_density()/subfind_density.c/161
  30 2992 0  	  DensDataResult     96.4245   27950.2258  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=30: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.7742 MB).
task 30: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 30
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 100 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  27 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  27 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  27 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  27 2973 0         FOF_PList    380.6759   18861.4721  fof_fof()/fof.c/249
  27 2974 0         FOF_GList    634.4598   19495.9320  fof_fof()/fof.c/337
  27 2975 0             Group     10.7703   19506.7022  fof_fof()/fof.c/380
  27 2976 0   DomainNodeIndex      0.3458   19507.0480  force_treeallocate()/forcetree.c/4800
  27 2977 0        Nodes_base   3737.4974   23244.5454  force_treeallocate()/forcetree.c/4803
  27 2978 0     Extnodes_base   3114.5811   26359.1265  force_treeallocate()/forcetree.c/4811
  27 2979 0          Nextnode    222.3007   26581.4273  force_treeallocate()/forcetree.c/4820
  27 2980 0            Father    221.9056   26803.3328  force_treeallocate()/forcetree.c/4827
  27 2981 0           Ngblist     63.4460   26866.7788  subfind_density()/subfind_density.c/78
  27 2982 0         Dist2list    126.8920   26993.6708  subfind_density()/subfind_density.c/79
 100 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 100 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 100 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 100 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 100 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 100 2973 0         FOF_PList    380.4518   18861.2480  fof_fof()/fof.c/249
 100 2974 0         FOF_GList    634.0863   19495.3344  fof_fof()/fof.c/337
 100 2975 0             Group     12.0610   19507.3954  fof_fof()/fof.c/380
 100 2976 0   DomainNodeIndex      0.3458   19507.7412  force_treeallocate()/forcetree.c/4800
 100 2977 0        Nodes_base   3737.4974   23245.2385  force_treeallocate()/forcetree.c/4803
 100 2978 0     Extnodes_base   3114.5811   26359.8197  force_treeallocate()/forcetree.c/4811
 100 2979 0          Nextnode    222.3007   26582.1204  force_treeallocate()/forcetree.c/4820
  27 2983 0    DataIndexTable     21.8182   27015.4890  subfind_density()/subfind_density.c/87
  27 2984 0      DataNodeList     58.1818   27073.6708  subfind_density()/subfind_density.c/89
  27 2985 0              Left    126.8920   27200.5627  subfind_density()/subfind_density.c/91
  27 2986 0             Right    126.8920   27327.4547  subfind_density()/subfind_density.c/92
 100 2980 0            Father    221.9056   26804.0260  force_treeallocate()/forcetree.c/4827
 100 2981 0           Ngblist     63.4086   26867.4347  subfind_density()/subfind_density.c/78
 100 2982 0         Dist2list    126.8173   26994.2519  subfind_density()/subfind_density.c/79
 100 2983 0    DataIndexTable     21.8182   27016.0701  subfind_density()/subfind_density.c/87
  27 2987 0              Todo     15.8615   27343.3162  subfind_density()/subfind_density.c/93
  27 2988 0             DM_Vx    126.8920   27470.2082  subfind_density()/subfind_density.c/95
  27 2989 0             DM_Vy    126.8920   27597.1001  subfind_density()/subfind_density.c/96
 100 2984 0      DataNodeList     58.1818   27074.2519  subfind_density()/subfind_density.c/89
 100 2985 0              Left    126.8173   27201.0692  subfind_density()/subfind_density.c/91
 100 2986 0             Right    126.8173   27327.8864  subfind_density()/subfind_density.c/92
  27 2990 0             DM_Vz    126.8920   27723.9921  subfind_density()/subfind_density.c/97
  27 2991 0    	  DensDataGet    124.7986   27848.7907  subfind_density()/subfind_density.c/161
  27 2992 0  	  DensDataResult     93.5989   27942.3896  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 100 2987 0              Todo     15.8522   27343.7386  subfind_density()/subfind_density.c/93
 100 2988 0             DM_Vx    126.8173   27470.5559  subfind_density()/subfind_density.c/95
 100 2989 0             DM_Vy    126.8173   27597.3731  subfind_density()/subfind_density.c/96

Task=27: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.6104 MB).
task 27: endrun called with an error level of 812


 100 2990 0             DM_Vz    126.8173   27724.1904  subfind_density()/subfind_density.c/97
 100 2991 0    	  DensDataGet    133.6901   27857.8805  subfind_density()/subfind_density.c/161
 100 2992 0  	  DensDataResult    100.2675   27958.1480  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 27

Task=100: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=41.852 MB).
task 100: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 100
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  96 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  96 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  96 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  96 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  96 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  96 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  96 2973 0         FOF_PList    380.4165   18861.2127  fof_fof()/fof.c/249
  96 2974 0         FOF_GList    634.0275   19495.2402  fof_fof()/fof.c/337
  96 2975 0             Group     12.7168   19507.9570  fof_fof()/fof.c/380
  31 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  31 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  31 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  31 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  31 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  31 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  31 2973 0         FOF_PList    380.4188   18861.2150  fof_fof()/fof.c/249
  31 2974 0         FOF_GList    634.0313   19495.2464  fof_fof()/fof.c/337
  31 2975 0             Group     12.2571   19507.5035  fof_fof()/fof.c/380
  31 2976 0   DomainNodeIndex      0.3458   19507.8493  force_treeallocate()/forcetree.c/4800
  96 2976 0   DomainNodeIndex      0.3458   19508.3028  force_treeallocate()/forcetree.c/4800
  96 2977 0        Nodes_base   3737.4974   23245.8001  force_treeallocate()/forcetree.c/4803
  96 2978 0     Extnodes_base   3114.5811   26360.3813  force_treeallocate()/forcetree.c/4811
  96 2979 0          Nextnode    222.3007   26582.6820  force_treeallocate()/forcetree.c/4820
  31 2977 0        Nodes_base   3737.4974   23245.3466  force_treeallocate()/forcetree.c/4803
  31 2978 0     Extnodes_base   3114.5811   26359.9278  force_treeallocate()/forcetree.c/4811
  31 2979 0          Nextnode    222.3007   26582.2285  force_treeallocate()/forcetree.c/4820
  96 2980 0            Father    221.9056   26804.5876  force_treeallocate()/forcetree.c/4827
  96 2981 0           Ngblist     63.4027   26867.9903  subfind_density()/subfind_density.c/78
  96 2982 0         Dist2list    126.8055   26994.7958  subfind_density()/subfind_density.c/79
  31 2980 0            Father    221.9056   26804.1341  force_treeallocate()/forcetree.c/4827
  31 2981 0           Ngblist     63.4031   26867.5372  subfind_density()/subfind_density.c/78
  31 2982 0         Dist2list    126.8063   26994.3435  subfind_density()/subfind_density.c/79
  96 2983 0    DataIndexTable     21.8182   27016.6140  subfind_density()/subfind_density.c/87
  96 2984 0      DataNodeList     58.1818   27074.7958  subfind_density()/subfind_density.c/89
  96 2985 0              Left    126.8055   27201.6013  subfind_density()/subfind_density.c/91
  31 2983 0    DataIndexTable     21.8182   27016.1617  subfind_density()/subfind_density.c/87
  31 2984 0      DataNodeList     58.1818   27074.3435  subfind_density()/subfind_density.c/89
  31 2985 0              Left    126.8063   27201.1497  subfind_density()/subfind_density.c/91
  96 2986 0             Right    126.8055   27328.4068  subfind_density()/subfind_density.c/92
  96 2987 0              Todo     15.8507   27344.2575  subfind_density()/subfind_density.c/93
  96 2988 0             DM_Vx    126.8055   27471.0630  subfind_density()/subfind_density.c/95
  31 2986 0             Right    126.8063   27327.9560  subfind_density()/subfind_density.c/92
  31 2987 0              Todo     15.8508   27343.8068  subfind_density()/subfind_density.c/93
  31 2988 0             DM_Vx    126.8063   27470.6131  subfind_density()/subfind_density.c/95
  31 2989 0             DM_Vy    126.8063   27597.4193  subfind_density()/subfind_density.c/96
  96 2989 0             DM_Vy    126.8055   27597.8685  subfind_density()/subfind_density.c/96
  31 2990 0             DM_Vz    126.8063   27724.2256  subfind_density()/subfind_density.c/97
  31 2991 0    	  DensDataGet    126.7665   27850.9921  subfind_density()/subfind_density.c/161
  96 2990 0             DM_Vz    126.8055   27724.6740  subfind_density()/subfind_density.c/97
  31 2992 0  	  DensDataResult     95.0749   27946.0670  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=31: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=53.933 MB).
task 31: endrun called with an error level of 812


  96 2991 0    	  DensDataGet    115.4874   27840.1614  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 31
  96 2992 0  	  DensDataResult     86.6155   27926.7769  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=96: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.2231 MB).
task 96: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 96
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 96:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
task 96: endrun called with an error level of 124


    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 103 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 103 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 103 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 103 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 103 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 103 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 103 2973 0         FOF_PList    380.2991   18861.0953  fof_fof()/fof.c/249
 103 2974 0         FOF_GList    633.8318   19494.9272  fof_fof()/fof.c/337
 103 2975 0             Group     10.6924   19505.6196  fof_fof()/fof.c/380
 103 2976 0   DomainNodeIndex      0.3458   19505.9654  force_treeallocate()/forcetree.c/4800
 103 2977 0        Nodes_base   3737.4974   23243.4627  force_treeallocate()/forcetree.c/4803
 103 2978 0     Extnodes_base   3114.5811   26358.0439  force_treeallocate()/forcetree.c/4811
 103 2979 0          Nextnode    222.3007   26580.3446  force_treeallocate()/forcetree.c/4820
 103 2980 0            Father    221.9056   26802.2502  force_treeallocate()/forcetree.c/4827
 103 2981 0           Ngblist     63.3832   26865.6334  subfind_density()/subfind_density.c/78
 103 2982 0         Dist2list    126.7664   26992.3997  subfind_density()/subfind_density.c/79
 103 2983 0    DataIndexTable     21.8182   27014.2179  subfind_density()/subfind_density.c/87
 103 2984 0      DataNodeList     58.1818   27072.3997  subfind_density()/subfind_density.c/89
 103 2985 0              Left    126.7664   27199.1661  subfind_density()/subfind_density.c/91
 103 2986 0             Right    126.7664   27325.9324  subfind_density()/subfind_density.c/92
 103 2987 0              Todo     15.8458   27341.7783  subfind_density()/subfind_density.c/93
 103 2988 0             DM_Vx    126.7664   27468.5446  subfind_density()/subfind_density.c/95
 103 2989 0             DM_Vy    126.7664   27595.3110  subfind_density()/subfind_density.c/96
 103 2990 0             DM_Vz    126.7664   27722.0773  subfind_density()/subfind_density.c/97
 103 2991 0    	  DensDataGet    114.6025   27836.6798  subfind_density()/subfind_density.c/161
 103 2992 0  	  DensDataResult     85.9519   27922.6317  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=103: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.3683 MB).
task 103: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 103
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 123 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 123 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 123 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 123 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   4 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   4 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   4 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   4 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   4 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   4 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
   4 2973 0         FOF_PList    380.4961   18861.2924  fof_fof()/fof.c/249
   4 2974 0         FOF_GList    634.1602   19495.4526  fof_fof()/fof.c/337
   4 2975 0             Group     11.0732   19506.5258  fof_fof()/fof.c/380
   4 2976 0   DomainNodeIndex      0.3458   19506.8716  force_treeallocate()/forcetree.c/4800
 123 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 123 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 123 2973 0         FOF_PList    380.4230   18861.2193  fof_fof()/fof.c/249
 123 2974 0         FOF_GList    634.0384   19495.2577  fof_fof()/fof.c/337
 123 2975 0             Group     12.4605   19507.7182  fof_fof()/fof.c/380
 123 2976 0   DomainNodeIndex      0.3458   19508.0639  force_treeallocate()/forcetree.c/4800
 123 2977 0        Nodes_base   3737.4974   23245.5613  force_treeallocate()/forcetree.c/4803
 123 2978 0     Extnodes_base   3114.5811   26360.1424  force_treeallocate()/forcetree.c/4811
 123 2979 0          Nextnode    222.3007   26582.4432  force_treeallocate()/forcetree.c/4820
 123 2980 0            Father    221.9056   26804.3488  force_treeallocate()/forcetree.c/4827
 123 2981 0           Ngblist     63.4038   26867.7526  subfind_density()/subfind_density.c/78
 123 2982 0         Dist2list    126.8077   26994.5603  subfind_density()/subfind_density.c/79
 123 2983 0    DataIndexTable     21.8182   27016.3785  subfind_density()/subfind_density.c/87
 123 2984 0      DataNodeList     58.1818   27074.5603  subfind_density()/subfind_density.c/89
 123 2985 0              Left    126.8077   27201.3679  subfind_density()/subfind_density.c/91
 123 2986 0             Right    126.8077   27328.1756  subfind_density()/subfind_density.c/92
 123 2987 0              Todo     15.8510   27344.0266  subfind_density()/subfind_density.c/93
 123 2988 0             DM_Vx    126.8077   27470.8343  subfind_density()/subfind_density.c/95
 123 2989 0             DM_Vy    126.8077   27597.6419  subfind_density()/subfind_density.c/96
   4 2977 0        Nodes_base   3737.4974   23244.3690  force_treeallocate()/forcetree.c/4803
   4 2978 0     Extnodes_base   3114.5811   26358.9501  force_treeallocate()/forcetree.c/4811
   4 2979 0          Nextnode    222.3007   26581.2509  force_treeallocate()/forcetree.c/4820
   4 2980 0            Father    221.9056   26803.1564  force_treeallocate()/forcetree.c/4827
 123 2990 0             DM_Vz    126.8077   27724.4496  subfind_density()/subfind_density.c/97
 123 2991 0    	  DensDataGet    113.5509   27838.0005  subfind_density()/subfind_density.c/161
 123 2992 0  	  DensDataResult     85.1632   27923.1637  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=123: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.8363 MB).
task 123: endrun called with an error level of 812


   4 2981 0           Ngblist     63.4160   26866.5725  subfind_density()/subfind_density.c/78
   4 2982 0         Dist2list    126.8320   26993.4045  subfind_density()/subfind_density.c/79
   4 2983 0    DataIndexTable     21.8182   27015.2227  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 123
   4 2984 0      DataNodeList     58.1818   27073.4045  subfind_density()/subfind_density.c/89
   4 2985 0              Left    126.8320   27200.2365  subfind_density()/subfind_density.c/91
   4 2986 0             Right    126.8320   27327.0686  subfind_density()/subfind_density.c/92
   4 2987 0              Todo     15.8540   27342.9226  subfind_density()/subfind_density.c/93
   4 2988 0             DM_Vx    126.8320   27469.7546  subfind_density()/subfind_density.c/95
   4 2989 0             DM_Vy    126.8320   27596.5867  subfind_density()/subfind_density.c/96
   4 2990 0             DM_Vz    126.8320   27723.4187  subfind_density()/subfind_density.c/97
   4 2991 0    	  DensDataGet    131.3467   27854.7655  subfind_density()/subfind_density.c/161
   4 2992 0  	  DensDataResult     98.5101   27953.2755  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=4: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.7245 MB).
task 4: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 4
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  19 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  19 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  19 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  19 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  19 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  19 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  19 2973 0         FOF_PList    380.4718   18861.2680  fof_fof()/fof.c/249
  19 2974 0         FOF_GList    634.1197   19495.3877  fof_fof()/fof.c/337
  19 2975 0             Group     11.2911   19506.6789  fof_fof()/fof.c/380
  19 2976 0   DomainNodeIndex      0.3458   19507.0246  force_treeallocate()/forcetree.c/4800
  19 2977 0        Nodes_base   3737.4974   23244.5220  force_treeallocate()/forcetree.c/4803
  19 2978 0     Extnodes_base   3114.5811   26359.1031  force_treeallocate()/forcetree.c/4811
  19 2979 0          Nextnode    222.3007   26581.4039  force_treeallocate()/forcetree.c/4820
  19 2980 0            Father    221.9056   26803.3095  force_treeallocate()/forcetree.c/4827
  19 2981 0           Ngblist     63.4120   26866.7214  subfind_density()/subfind_density.c/78
  19 2982 0         Dist2list    126.8239   26993.5454  subfind_density()/subfind_density.c/79
  19 2983 0    DataIndexTable     21.8182   27015.3635  subfind_density()/subfind_density.c/87
  42 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  42 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  42 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  42 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  42 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  42 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  42 2973 0         FOF_PList    380.6977   18861.4939  fof_fof()/fof.c/249
  42 2974 0         FOF_GList    634.4962   19495.9901  fof_fof()/fof.c/337
  42 2975 0             Group     10.5897   19506.5798  fof_fof()/fof.c/380
 108 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 108 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 108 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 108 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 108 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 108 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  19 2984 0      DataNodeList     58.1818   27073.5453  subfind_density()/subfind_density.c/89
  19 2985 0              Left    126.8239   27200.3693  subfind_density()/subfind_density.c/91
  19 2986 0             Right    126.8239   27327.1932  subfind_density()/subfind_density.c/92
  19 2987 0              Todo     15.8530   27343.0462  subfind_density()/subfind_density.c/93
  19 2988 0             DM_Vx    126.8239   27469.8701  subfind_density()/subfind_density.c/95
  19 2989 0             DM_Vy    126.8239   27596.6941  subfind_density()/subfind_density.c/96
  19 2990 0             DM_Vz    126.8239   27723.5180  subfind_density()/subfind_density.c/97
  42 2976 0   DomainNodeIndex      0.3458   19506.9256  force_treeallocate()/forcetree.c/4800
  42 2977 0        Nodes_base   3737.4974   23244.4229  force_treeallocate()/forcetree.c/4803
  42 2978 0     Extnodes_base   3114.5811   26359.0041  force_treeallocate()/forcetree.c/4811
 108 2973 0         FOF_PList    379.9593   18860.7555  fof_fof()/fof.c/249
 108 2974 0         FOF_GList    633.2655   19494.0210  fof_fof()/fof.c/337
  19 2991 0    	  DensDataGet    129.6235   27853.1416  subfind_density()/subfind_density.c/161
  19 2992 0  	  DensDataResult     97.2177   27950.3592  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=19: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.6408 MB).
  42 2979 0          Nextnode    222.3007   26581.3048  force_treeallocate()/forcetree.c/4820
  42 2980 0            Father    221.9056   26803.2104  force_treeallocate()/forcetree.c/4827
 108 2975 0             Group     12.3983   19506.4192  fof_fof()/fof.c/380
 108 2976 0   DomainNodeIndex      0.3458   19506.7650  force_treeallocate()/forcetree.c/4800
 108 2977 0        Nodes_base   3737.4974   23244.2624  force_treeallocate()/forcetree.c/4803
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
task 19: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  42 2981 0           Ngblist     63.4496   26866.6600  subfind_density()/subfind_density.c/78
  42 2982 0         Dist2list    126.8992   26993.5592  subfind_density()/subfind_density.c/79
  42 2983 0    DataIndexTable     21.8182   27015.3774  subfind_density()/subfind_density.c/87
 108 2978 0     Extnodes_base   3114.5811   26358.8435  force_treeallocate()/forcetree.c/4811
 108 2979 0          Nextnode    222.3007   26581.1442  force_treeallocate()/forcetree.c/4820
  87 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  87 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  87 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  87 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  87 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  87 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  87 2973 0         FOF_PList    380.3392   18861.1355  fof_fof()/fof.c/249
  87 2974 0         FOF_GList    633.8987   19495.0342  fof_fof()/fof.c/337
  87 2975 0             Group     12.3651   19507.3993  fof_fof()/fof.c/380
  87 2976 0   DomainNodeIndex      0.3458   19507.7450  force_treeallocate()/forcetree.c/4800
  87 2977 0        Nodes_base   3737.4974   23245.2424  force_treeallocate()/forcetree.c/4803
  87 2978 0     Extnodes_base   3114.5811   26359.8235  force_treeallocate()/forcetree.c/4811
  87 2979 0          Nextnode    222.3007   26582.1243  force_treeallocate()/forcetree.c/4820
  87 2980 0            Father    221.9056   26804.0299  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 19
  40 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  40 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  40 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  40 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  40 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  40 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  40 2973 0         FOF_PList    380.4093   18861.2056  fof_fof()/fof.c/249
  42 2984 0      DataNodeList     58.1818   27073.5592  subfind_density()/subfind_density.c/89
  42 2985 0              Left    126.8992   27200.4585  subfind_density()/subfind_density.c/91
  42 2986 0             Right    126.8992   27327.3577  subfind_density()/subfind_density.c/92
 108 2980 0            Father    221.9056   26803.0498  force_treeallocate()/forcetree.c/4827
 108 2981 0           Ngblist     63.3265   26866.3764  subfind_density()/subfind_density.c/78
 108 2982 0         Dist2list    126.6531   26993.0295  subfind_density()/subfind_density.c/79
 108 2983 0    DataIndexTable     21.8182   27014.8476  subfind_density()/subfind_density.c/87
 108 2984 0      DataNodeList     58.1818   27073.0294  subfind_density()/subfind_density.c/89
 108 2985 0              Left    126.6531   27199.6825  subfind_density()/subfind_density.c/91
 108 2986 0             Right    126.6531   27326.3356  subfind_density()/subfind_density.c/92
  87 2981 0           Ngblist     63.3899   26867.4197  subfind_density()/subfind_density.c/78
  87 2982 0         Dist2list    126.7797   26994.1995  subfind_density()/subfind_density.c/79
  87 2983 0    DataIndexTable     21.8182   27016.0177  subfind_density()/subfind_density.c/87
  87 2984 0      DataNodeList     58.1818   27074.1995  subfind_density()/subfind_density.c/89
  40 2974 0         FOF_GList    634.0155   19495.2211  fof_fof()/fof.c/337
  40 2975 0             Group     12.1503   19507.3714  fof_fof()/fof.c/380
  42 2987 0              Todo     15.8624   27343.2201  subfind_density()/subfind_density.c/93
 108 2987 0              Todo     15.8316   27342.1673  subfind_density()/subfind_density.c/93
 108 2988 0             DM_Vx    126.6531   27468.8203  subfind_density()/subfind_density.c/95
  87 2985 0              Left    126.7797   27200.9792  subfind_density()/subfind_density.c/91
  87 2986 0             Right    126.7797   27327.7589  subfind_density()/subfind_density.c/92
  87 2987 0              Todo     15.8475   27343.6064  subfind_density()/subfind_density.c/93
  40 2976 0   DomainNodeIndex      0.3458   19507.7171  force_treeallocate()/forcetree.c/4800
  40 2977 0        Nodes_base   3737.4974   23245.2145  force_treeallocate()/forcetree.c/4803
  42 2988 0             DM_Vx    126.8992   27470.1193  subfind_density()/subfind_density.c/95
 108 2989 0             DM_Vy    126.6531   27595.4734  subfind_density()/subfind_density.c/96
  87 2988 0             DM_Vx    126.7797   27470.3862  subfind_density()/subfind_density.c/95
  87 2989 0             DM_Vy    126.7797   27597.1659  subfind_density()/subfind_density.c/96
  87 2990 0             DM_Vz    126.7797   27723.9457  subfind_density()/subfind_density.c/97
  40 2978 0     Extnodes_base   3114.5811   26359.7956  force_treeallocate()/forcetree.c/4811
  40 2979 0          Nextnode    222.3007   26582.0964  force_treeallocate()/forcetree.c/4820
  40 2980 0            Father    221.9056   26804.0020  force_treeallocate()/forcetree.c/4827
  42 2989 0             DM_Vy    126.8992   27597.0185  subfind_density()/subfind_density.c/96
 108 2990 0             DM_Vz    126.6531   27722.1265  subfind_density()/subfind_density.c/97
  87 2991 0    	  DensDataGet    109.0040   27832.9496  subfind_density()/subfind_density.c/161
  87 2992 0  	  DensDataResult     81.7530   27914.7026  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=87: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.2974 MB).
task 87: endrun called with an error level of 812


  40 2981 0           Ngblist     63.4016   26867.4035  subfind_density()/subfind_density.c/78
  40 2982 0         Dist2list    126.8031   26994.2066  subfind_density()/subfind_density.c/79
  42 2990 0             DM_Vz    126.8992   27723.9178  subfind_density()/subfind_density.c/97
 108 2991 0    	  DensDataGet    111.0378   27833.1643  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 87
  40 2983 0    DataIndexTable     21.8182   27016.0248  subfind_density()/subfind_density.c/87
  40 2984 0      DataNodeList     58.1818   27074.2066  subfind_density()/subfind_density.c/89
  40 2985 0              Left    126.8031   27201.0097  subfind_density()/subfind_density.c/91
  40 2986 0             Right    126.8031   27327.8128  subfind_density()/subfind_density.c/92
  40 2987 0              Todo     15.8504   27343.6632  subfind_density()/subfind_density.c/93
  42 2991 0    	  DensDataGet    110.2749   27834.1927  subfind_density()/subfind_density.c/161
 108 2992 0  	  DensDataResult     83.2783   27916.4426  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  40 2988 0             DM_Vx    126.8031   27470.4663  subfind_density()/subfind_density.c/95
  42 2992 0  	  DensDataResult     82.7062   27916.8989  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

  40 2989 0             DM_Vy    126.8031   27597.2694  subfind_density()/subfind_density.c/96

Task=42: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.1011 MB).

Task=108: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.5574 MB).
task 108: endrun called with an error level of 812


  40 2990 0             DM_Vz    126.8031   27724.0725  subfind_density()/subfind_density.c/97
task 42: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 108
  40 2991 0    	  DensDataGet    135.8300   27859.9025  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 42
  40 2992 0  	  DensDataResult    101.8725   27961.7750  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=40: Not enough memory in mymalloc_fullinfo() to allocate 87.2725 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=38.225 MB).
task 40: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 40
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 An HDF5 error was detected, we'd better stop.
  86 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  86 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  86 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  86 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  86 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
HDF5-DIAG: Error detected in HDF5 (1.8.12)   86 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  86 2973 0         FOF_PList    380.5031   18861.2993  fof_fof()/fof.c/249
  86 2974 0         FOF_GList    634.1718   19495.4711  fof_fof()/fof.c/337
MPI-process 40:
  86 2975 0             Group     11.4758   19506.9469  fof_fof()/fof.c/380
  86 2976 0   DomainNodeIndex      0.3458   19507.2927  force_treeallocate()/forcetree.c/4800
task 40: endrun called with an error level of 124


  86 2977 0        Nodes_base   3737.4974   23244.7901  force_treeallocate()/forcetree.c/4803
  86 2978 0     Extnodes_base   3114.5811   26359.3712  force_treeallocate()/forcetree.c/4811
  86 2979 0          Nextnode    222.3007   26581.6719  force_treeallocate()/forcetree.c/4820
  86 2980 0            Father    221.9056   26803.5775  force_treeallocate()/forcetree.c/4827
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 40
  86 2981 0           Ngblist     63.4172   26866.9947  subfind_density()/subfind_density.c/78
  86 2982 0         Dist2list    126.8344   26993.8291  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  86 2983 0    DataIndexTable     21.8182   27015.6472  subfind_density()/subfind_density.c/87
  86 2984 0      DataNodeList     58.1818   27073.8290  subfind_density()/subfind_density.c/89
  86 2985 0              Left    126.8344   27200.6634  subfind_density()/subfind_density.c/91
 121 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 121 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 121 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 121 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 121 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 121 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 121 2973 0         FOF_PList    380.5883   18861.3846  fof_fof()/fof.c/249
 121 2974 0         FOF_GList    634.3139   19495.6985  fof_fof()/fof.c/337
 121 2975 0             Group     12.4501   19508.1486  fof_fof()/fof.c/380
 121 2976 0   DomainNodeIndex      0.3458   19508.4943  force_treeallocate()/forcetree.c/4800
 121 2977 0        Nodes_base   3737.4974   23245.9917  force_treeallocate()/forcetree.c/4803
 121 2978 0     Extnodes_base   3114.5811   26360.5729  force_treeallocate()/forcetree.c/4811
 121 2979 0          Nextnode    222.3007   26582.8736  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  86 2986 0             Right    126.8344   27327.4978  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  41 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  41 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  41 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  41 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  41 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  41 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  41 2973 0         FOF_PList    379.9692   18860.7655  fof_fof()/fof.c/249
  41 2974 0         FOF_GList    633.2821   19494.0475  fof_fof()/fof.c/337
  41 2975 0             Group     10.9052   19504.9527  fof_fof()/fof.c/380
 121 2980 0            Father    221.9056   26804.7792  force_treeallocate()/forcetree.c/4827
 121 2981 0           Ngblist     63.4314   26868.2106  subfind_density()/subfind_density.c/78
 121 2982 0         Dist2list    126.8628   26995.0734  subfind_density()/subfind_density.c/79
 121 2983 0    DataIndexTable     21.8182   27016.8915  subfind_density()/subfind_density.c/87
   5 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   5 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   5 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   5 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   5 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   5 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
   5 2973 0         FOF_PList    380.5985   18861.3948  fof_fof()/fof.c/249
   5 2974 0         FOF_GList    634.3309   19495.7257  fof_fof()/fof.c/337
   5 2975 0             Group     11.5464   19507.2721  fof_fof()/fof.c/380
   5 2976 0   DomainNodeIndex      0.3458   19507.6178  force_treeallocate()/forcetree.c/4800
   5 2977 0        Nodes_base   3737.4974   23245.1152  force_treeallocate()/forcetree.c/4803
  86 2987 0              Todo     15.8543   27343.3521  subfind_density()/subfind_density.c/93
 122 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 122 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 122 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 122 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 122 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 122 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 122 2973 0         FOF_PList    380.6480   18861.4442  fof_fof()/fof.c/249
 122 2974 0         FOF_GList    634.4133   19495.8575  fof_fof()/fof.c/337
 122 2975 0             Group     13.1858   19509.0432  fof_fof()/fof.c/380
  41 2976 0   DomainNodeIndex      0.3458   19505.2984  force_treeallocate()/forcetree.c/4800
  41 2977 0        Nodes_base   3737.4974   23242.7958  force_treeallocate()/forcetree.c/4803
  41 2978 0     Extnodes_base   3114.5811   26357.3769  force_treeallocate()/forcetree.c/4811
 121 2984 0      DataNodeList     58.1818   27075.0733  subfind_density()/subfind_density.c/89
 121 2985 0              Left    126.8628   27201.9361  subfind_density()/subfind_density.c/91
 121 2986 0             Right    126.8628   27328.7989  subfind_density()/subfind_density.c/92
   5 2978 0     Extnodes_base   3114.5811   26359.6963  force_treeallocate()/forcetree.c/4811
   5 2979 0          Nextnode    222.3007   26581.9971  force_treeallocate()/forcetree.c/4820
   5 2980 0            Father    221.9056   26803.9027  force_treeallocate()/forcetree.c/4827
   5 2981 0           Ngblist     63.4331   26867.3358  subfind_density()/subfind_density.c/78
  86 2988 0             DM_Vx    126.8344   27470.1864  subfind_density()/subfind_density.c/95
 122 2976 0   DomainNodeIndex      0.3458   19509.3890  force_treeallocate()/forcetree.c/4800
 122 2977 0        Nodes_base   3737.4974   23246.8864  force_treeallocate()/forcetree.c/4803
 122 2978 0     Extnodes_base   3114.5811   26361.4675  force_treeallocate()/forcetree.c/4811
  41 2979 0          Nextnode    222.3007   26579.6777  force_treeallocate()/forcetree.c/4820
 121 2987 0              Todo     15.8578   27344.6567  subfind_density()/subfind_density.c/93
 121 2988 0             DM_Vx    126.8628   27471.5195  subfind_density()/subfind_density.c/95
   5 2982 0         Dist2list    126.8662   26994.2019  subfind_density()/subfind_density.c/79
   5 2983 0    DataIndexTable     21.8182   27016.0201  subfind_density()/subfind_density.c/87
   5 2984 0      DataNodeList     58.1818   27074.2019  subfind_density()/subfind_density.c/89
  86 2989 0             DM_Vy    126.8344   27597.0208  subfind_density()/subfind_density.c/96
 122 2979 0          Nextnode    222.3007   26583.7683  force_treeallocate()/forcetree.c/4820
 122 2980 0            Father    221.9056   26805.6738  force_treeallocate()/forcetree.c/4827
 122 2981 0           Ngblist     63.4413   26869.1152  subfind_density()/subfind_density.c/78
  41 2980 0            Father    221.9056   26801.5833  force_treeallocate()/forcetree.c/4827
 121 2989 0             DM_Vy    126.8628   27598.3823  subfind_density()/subfind_density.c/96
 121 2990 0             DM_Vz    126.8628   27725.2451  subfind_density()/subfind_density.c/97
   5 2985 0              Left    126.8662   27201.0681  subfind_density()/subfind_density.c/91
   5 2986 0             Right    126.8662   27327.9343  subfind_density()/subfind_density.c/92
   5 2987 0              Todo     15.8583   27343.7925  subfind_density()/subfind_density.c/93
   5 2988 0             DM_Vx    126.8662   27470.6587  subfind_density()/subfind_density.c/95
  86 2990 0             DM_Vz    126.8344   27723.8551  subfind_density()/subfind_density.c/97
 122 2982 0         Dist2list    126.8827   26995.9978  subfind_density()/subfind_density.c/79
 122 2983 0    DataIndexTable     21.8182   27017.8160  subfind_density()/subfind_density.c/87
 122 2984 0      DataNodeList     58.1818   27075.9978  subfind_density()/subfind_density.c/89
  41 2981 0           Ngblist     63.3282   26864.9115  subfind_density()/subfind_density.c/78
 121 2991 0    	  DensDataGet    143.9434   27869.1884  subfind_density()/subfind_density.c/161
 121 2992 0  	  DensDataResult    107.9575   27977.1459  subfind_density()/subfind_density.c/201
   5 2989 0             DM_Vy    126.8662   27597.5249  subfind_density()/subfind_density.c/96
   5 2990 0             DM_Vz    126.8662   27724.3911  subfind_density()/subfind_density.c/97
  86 2991 0    	  DensDataGet    130.9791   27854.8343  subfind_density()/subfind_density.c/161
 122 2985 0              Left    126.8827   27202.8804  subfind_density()/subfind_density.c/91
 122 2986 0             Right    126.8827   27329.7631  subfind_density()/subfind_density.c/92
 122 2987 0              Todo     15.8603   27345.6234  subfind_density()/subfind_density.c/93
  41 2982 0         Dist2list    126.6564   26991.5679  subfind_density()/subfind_density.c/79
----------------------------------------------------------------------------------------
   5 2991 0    	  DensDataGet    130.0559   27854.4470  subfind_density()/subfind_density.c/161
   5 2992 0  	  DensDataResult     97.5419   27951.9889  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  86 2992 0  	  DensDataResult     98.2343   27953.0686  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 122 2988 0             DM_Vx    126.8827   27472.5061  subfind_density()/subfind_density.c/95
 122 2989 0             DM_Vy    126.8827   27599.3887  subfind_density()/subfind_density.c/96
  41 2983 0    DataIndexTable     21.8182   27013.3861  subfind_density()/subfind_density.c/87

Task=121: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=22.8541 MB).
task 121: endrun called with an error level of 812



Task=5: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=48.0111 MB).
task 5: endrun called with an error level of 812



Task=86: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.9314 MB).
task 86: endrun called with an error level of 812


 122 2990 0             DM_Vz    126.8827   27726.2714  subfind_density()/subfind_density.c/97
  41 2984 0      DataNodeList     58.1818   27071.5679  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 121
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 5
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 86
 122 2991 0    	  DensDataGet    132.9633   27859.2346  subfind_density()/subfind_density.c/161
  41 2985 0              Left    126.6564   27198.2243  subfind_density()/subfind_density.c/91
 122 2992 0  	  DensDataResult     99.7224   27958.9571  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  41 2986 0             Right    126.6564   27324.8807  subfind_density()/subfind_density.c/92

Task=122: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=41.0429 MB).
task 122: endrun called with an error level of 812


  41 2987 0              Todo     15.8321   27340.7127  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 122
  41 2988 0             DM_Vx    126.6564   27467.3691  subfind_density()/subfind_density.c/95
  41 2989 0             DM_Vy    126.6564   27594.0256  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  41 2990 0             DM_Vz    126.6564   27720.6820  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  47 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  47 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  47 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  47 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  47 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  47 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  47 2973 0         FOF_PList    380.5895   18861.3858  fof_fof()/fof.c/249
  47 2974 0         FOF_GList    634.3159   19495.7017  fof_fof()/fof.c/337
  47 2975 0             Group     11.9583   19507.6600  fof_fof()/fof.c/380
  47 2976 0   DomainNodeIndex      0.3458   19508.0058  force_treeallocate()/forcetree.c/4800
  47 2977 0        Nodes_base   3737.4974   23245.5031  force_treeallocate()/forcetree.c/4803
  47 2978 0     Extnodes_base   3114.5811   26360.0843  force_treeallocate()/forcetree.c/4811
  47 2979 0          Nextnode    222.3007   26582.3850  force_treeallocate()/forcetree.c/4820
  47 2980 0            Father    221.9056   26804.2906  force_treeallocate()/forcetree.c/4827
  47 2981 0           Ngblist     63.4316   26867.7222  subfind_density()/subfind_density.c/78
  41 2991 0    	  DensDataGet    119.0554   27839.7373  subfind_density()/subfind_density.c/161
  80 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  80 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  80 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  80 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  80 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  80 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  80 2973 0         FOF_PList    380.4087   18861.2050  fof_fof()/fof.c/249
  80 2974 0         FOF_GList    634.0145   19495.2195  fof_fof()/fof.c/337
  80 2975 0             Group     11.1179   19506.3374  fof_fof()/fof.c/380
  80 2976 0   DomainNodeIndex      0.3458   19506.6831  force_treeallocate()/forcetree.c/4800
  80 2977 0        Nodes_base   3737.4974   23244.1805  force_treeallocate()/forcetree.c/4803
  80 2978 0     Extnodes_base   3114.5811   26358.7617  force_treeallocate()/forcetree.c/4811
  47 2982 0         Dist2list    126.8632   26994.5854  subfind_density()/subfind_density.c/79
  47 2983 0    DataIndexTable     21.8182   27016.4036  subfind_density()/subfind_density.c/87
  47 2984 0      DataNodeList     58.1818   27074.5854  subfind_density()/subfind_density.c/89
  47 2985 0              Left    126.8632   27201.4485  subfind_density()/subfind_density.c/91
  47 2986 0             Right    126.8632   27328.3117  subfind_density()/subfind_density.c/92
  41 2992 0  	  DensDataResult     89.2915   27929.0288  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  80 2979 0          Nextnode    222.3007   26581.0624  force_treeallocate()/forcetree.c/4820
  80 2980 0            Father    221.9056   26802.9680  force_treeallocate()/forcetree.c/4827
  80 2981 0           Ngblist     63.4015   26866.3694  subfind_density()/subfind_density.c/78
  80 2982 0         Dist2list    126.8029   26993.1723  subfind_density()/subfind_density.c/79
  80 2983 0    DataIndexTable     21.8182   27014.9905  subfind_density()/subfind_density.c/87
  47 2987 0              Todo     15.8579   27344.1696  subfind_density()/subfind_density.c/93
  47 2988 0             DM_Vx    126.8632   27471.0328  subfind_density()/subfind_density.c/95
  47 2989 0             DM_Vy    126.8632   27597.8960  subfind_density()/subfind_density.c/96

Task=41: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.9712 MB).
task 41: endrun called with an error level of 812


  80 2984 0      DataNodeList     58.1818   27073.1723  subfind_density()/subfind_density.c/89
  80 2985 0              Left    126.8029   27199.9752  subfind_density()/subfind_density.c/91
  80 2986 0             Right    126.8029   27326.7781  subfind_density()/subfind_density.c/92
  47 2990 0             DM_Vz    126.8632   27724.7592  subfind_density()/subfind_density.c/97
  47 2991 0    	  DensDataGet    137.8763   27862.6355  subfind_density()/subfind_density.c/161
  47 2992 0  	  DensDataResult    103.4073   27966.0428  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 41
  80 2987 0              Todo     15.8504   27342.6285  subfind_density()/subfind_density.c/93
  80 2988 0             DM_Vx    126.8029   27469.4314  subfind_density()/subfind_density.c/95
  80 2989 0             DM_Vy    126.8029   27596.2343  subfind_density()/subfind_density.c/96


Task=47: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=33.9572 MB).
task 47: endrun called with an error level of 812


  80 2990 0             DM_Vz    126.8029   27723.0372  subfind_density()/subfind_density.c/97
  80 2991 0    	  DensDataGet    124.1990   27847.2362  subfind_density()/subfind_density.c/161
  80 2992 0  	  DensDataResult     93.1492   27940.3854  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 47

Task=80: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.6146 MB).
task 80: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 80
task 80: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 80:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 80
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  44 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  44 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  44 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  44 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  44 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  44 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  44 2973 0         FOF_PList    380.6877   18861.4839  fof_fof()/fof.c/249
  44 2974 0         FOF_GList    634.4795   19495.9635  fof_fof()/fof.c/337
  44 2975 0             Group     12.2706   19508.2341  fof_fof()/fof.c/380
  44 2976 0   DomainNodeIndex      0.3458   19508.5799  force_treeallocate()/forcetree.c/4800
  44 2977 0        Nodes_base   3737.4974   23246.0772  force_treeallocate()/forcetree.c/4803
  44 2978 0     Extnodes_base   3114.5811   26360.6584  force_treeallocate()/forcetree.c/4811
  44 2979 0          Nextnode    222.3007   26582.9591  force_treeallocate()/forcetree.c/4820
  44 2980 0            Father    221.9056   26804.8647  force_treeallocate()/forcetree.c/4827
  44 2981 0           Ngblist     63.4480   26868.3127  subfind_density()/subfind_density.c/78
  44 2982 0         Dist2list    126.8959   26995.2086  subfind_density()/subfind_density.c/79
  44 2983 0    DataIndexTable     21.8182   27017.0267  subfind_density()/subfind_density.c/87
  44 2984 0      DataNodeList     58.1818   27075.2085  subfind_density()/subfind_density.c/89
  44 2985 0              Left    126.8959   27202.1044  subfind_density()/subfind_density.c/91
  44 2986 0             Right    126.8959   27329.0003  subfind_density()/subfind_density.c/92
  44 2987 0              Todo     15.8620   27344.8623  subfind_density()/subfind_density.c/93
  44 2988 0             DM_Vx    126.8959   27471.7582  subfind_density()/subfind_density.c/95
  44 2989 0             DM_Vy    126.8959   27598.6541  subfind_density()/subfind_density.c/96
  44 2990 0             DM_Vz    126.8959   27725.5500  subfind_density()/subfind_density.c/97
  44 2991 0    	  DensDataGet    113.1284   27838.6785  subfind_density()/subfind_density.c/161
  44 2992 0  	  DensDataResult     84.8463   27923.5248  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=44: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.4752 MB).
task 44: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 44
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  45 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  45 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  45 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  45 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  45 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  45 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  45 2973 0         FOF_PList    380.4509   18861.2471  fof_fof()/fof.c/249
  45 2974 0         FOF_GList    634.0848   19495.3319  fof_fof()/fof.c/337
  45 2975 0             Group     12.3173   19507.6492  fof_fof()/fof.c/380
  45 2976 0   DomainNodeIndex      0.3458   19507.9950  force_treeallocate()/forcetree.c/4800
  82 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  82 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  82 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  82 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  82 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  82 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  82 2973 0         FOF_PList    379.8411   18860.6374  fof_fof()/fof.c/249
  82 2974 0         FOF_GList    633.0686   19493.7060  fof_fof()/fof.c/337
  82 2975 0             Group     11.6750   19505.3810  fof_fof()/fof.c/380
  82 2976 0   DomainNodeIndex      0.3458   19505.7268  force_treeallocate()/forcetree.c/4800
  82 2977 0        Nodes_base   3737.4974   23243.2241  force_treeallocate()/forcetree.c/4803
  82 2978 0     Extnodes_base   3114.5811   26357.8053  force_treeallocate()/forcetree.c/4811
  82 2979 0          Nextnode    222.3007   26580.1060  force_treeallocate()/forcetree.c/4820
  45 2977 0        Nodes_base   3737.4974   23245.4923  force_treeallocate()/forcetree.c/4803
  45 2978 0     Extnodes_base   3114.5811   26360.0735  force_treeallocate()/forcetree.c/4811
  45 2979 0          Nextnode    222.3007   26582.3742  force_treeallocate()/forcetree.c/4820
  45 2980 0            Father    221.9056   26804.2798  force_treeallocate()/forcetree.c/4827
  82 2980 0            Father    221.9056   26802.0116  force_treeallocate()/forcetree.c/4827
  82 2981 0           Ngblist     63.3069   26865.3185  subfind_density()/subfind_density.c/78
  82 2982 0         Dist2list    126.6137   26991.9322  subfind_density()/subfind_density.c/79
  45 2981 0           Ngblist     63.4085   26867.6883  subfind_density()/subfind_density.c/78
  45 2982 0         Dist2list    126.8170   26994.5052  subfind_density()/subfind_density.c/79
  45 2983 0    DataIndexTable     21.8182   27016.3234  subfind_density()/subfind_density.c/87
  82 2983 0    DataIndexTable     21.8182   27013.7504  subfind_density()/subfind_density.c/87
  82 2984 0      DataNodeList     58.1818   27071.9322  subfind_density()/subfind_density.c/89
  82 2985 0              Left    126.6137   27198.5459  subfind_density()/subfind_density.c/91
  45 2984 0      DataNodeList     58.1818   27074.5052  subfind_density()/subfind_density.c/89
  45 2985 0              Left    126.8170   27201.3222  subfind_density()/subfind_density.c/91
  45 2986 0             Right    126.8170   27328.1391  subfind_density()/subfind_density.c/92
  45 2987 0              Todo     15.8521   27343.9912  subfind_density()/subfind_density.c/93
  82 2986 0             Right    126.6137   27325.1596  subfind_density()/subfind_density.c/92
  82 2987 0              Todo     15.8267   27340.9863  subfind_density()/subfind_density.c/93
  82 2988 0             DM_Vx    126.6137   27467.6000  subfind_density()/subfind_density.c/95
  82 2989 0             DM_Vy    126.6137   27594.2137  subfind_density()/subfind_density.c/96
  45 2988 0             DM_Vx    126.8170   27470.8082  subfind_density()/subfind_density.c/95
  45 2989 0             DM_Vy    126.8170   27597.6251  subfind_density()/subfind_density.c/96
  45 2990 0             DM_Vz    126.8170   27724.4421  subfind_density()/subfind_density.c/97
  82 2990 0             DM_Vz    126.6137   27720.8275  subfind_density()/subfind_density.c/97
  82 2991 0    	  DensDataGet    144.6204   27865.4478  subfind_density()/subfind_density.c/161
  45 2991 0    	  DensDataGet    139.3975   27863.8396  subfind_density()/subfind_density.c/161
  45 2992 0  	  DensDataResult    104.5481   27968.3878  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  82 2992 0  	  DensDataResult    108.4653   27973.9131  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=45: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=31.6122 MB).
task 45: endrun called with an error level of 812



Task=82: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=26.0869 MB).
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 45
task 82: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 82
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  46 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  46 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  46 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  46 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  46 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  46 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  46 2973 0         FOF_PList    380.4239   18861.2201  fof_fof()/fof.c/249
  46 2974 0         FOF_GList    634.0398   19495.2599  fof_fof()/fof.c/337
  81 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  81 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  81 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  81 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  81 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  81 2973 0         FOF_PList    380.4409   18861.2371  fof_fof()/fof.c/249
  81 2974 0         FOF_GList    634.0682   19495.3053  fof_fof()/fof.c/337
  81 2975 0             Group     10.9114   19506.2167  fof_fof()/fof.c/380
  81 2976 0   DomainNodeIndex      0.3458   19506.5625  force_treeallocate()/forcetree.c/4800
  81 2977 0        Nodes_base   3737.4974   23244.0598  force_treeallocate()/forcetree.c/4803
  81 2978 0     Extnodes_base   3114.5811   26358.6410  force_treeallocate()/forcetree.c/4811
  81 2979 0          Nextnode    222.3007   26580.9417  force_treeallocate()/forcetree.c/4820
  46 2975 0             Group     11.2974   19506.5573  fof_fof()/fof.c/380
  46 2976 0   DomainNodeIndex      0.3458   19506.9030  force_treeallocate()/forcetree.c/4800
  46 2977 0        Nodes_base   3737.4974   23244.4004  force_treeallocate()/forcetree.c/4803
  81 2980 0            Father    221.9056   26802.8473  force_treeallocate()/forcetree.c/4827
  81 2981 0           Ngblist     63.4068   26866.2541  subfind_density()/subfind_density.c/78
  81 2982 0         Dist2list    126.8136   26993.0678  subfind_density()/subfind_density.c/79
  81 2983 0    DataIndexTable     21.8182   27014.8859  subfind_density()/subfind_density.c/87
  46 2978 0     Extnodes_base   3114.5811   26358.9816  force_treeallocate()/forcetree.c/4811
  46 2979 0          Nextnode    222.3007   26581.2823  force_treeallocate()/forcetree.c/4820
  46 2980 0            Father    221.9056   26803.1879  force_treeallocate()/forcetree.c/4827
  81 2984 0      DataNodeList     58.1818   27073.0677  subfind_density()/subfind_density.c/89
  81 2985 0              Left    126.8136   27199.8814  subfind_density()/subfind_density.c/91
  81 2986 0             Right    126.8136   27326.6950  subfind_density()/subfind_density.c/92
  46 2981 0           Ngblist     63.4040   26866.5919  subfind_density()/subfind_density.c/78
  46 2982 0         Dist2list    126.8080   26993.3998  subfind_density()/subfind_density.c/79
  46 2983 0    DataIndexTable     21.8182   27015.2180  subfind_density()/subfind_density.c/87
  81 2987 0              Todo     15.8517   27342.5467  subfind_density()/subfind_density.c/93
  81 2988 0             DM_Vx    126.8136   27469.3604  subfind_density()/subfind_density.c/95
  81 2989 0             DM_Vy    126.8136   27596.1740  subfind_density()/subfind_density.c/96
  46 2984 0      DataNodeList     58.1818   27073.3998  subfind_density()/subfind_density.c/89
  46 2985 0              Left    126.8080   27200.2078  subfind_density()/subfind_density.c/91
  46 2986 0             Right    126.8080   27327.0157  subfind_density()/subfind_density.c/92
  81 2990 0             DM_Vz    126.8136   27722.9876  subfind_density()/subfind_density.c/97
  81 2991 0    	  DensDataGet    132.3776   27855.3652  subfind_density()/subfind_density.c/161
  81 2992 0  	  DensDataResult     99.2832   27954.6484  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  46 2987 0              Todo     15.8510   27342.8667  subfind_density()/subfind_density.c/93
  46 2988 0             DM_Vx    126.8080   27469.6747  subfind_density()/subfind_density.c/95


Task=81: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.3516 MB).
task 81: endrun called with an error level of 812


  46 2989 0             DM_Vy    126.8080   27596.4826  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 81
  46 2990 0             DM_Vz    126.8080   27723.2906  subfind_density()/subfind_density.c/97
  46 2991 0    	  DensDataGet    124.6417   27847.9323  subfind_density()/subfind_density.c/161
  46 2992 0  	  DensDataResult     93.4813   27941.4136  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=46: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.5864 MB).
task 46: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 46
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 120 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 120 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 120 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 120 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 112 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 112 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 112 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 112 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 112 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 112 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 112 2973 0         FOF_PList    380.2695   18861.0658  fof_fof()/fof.c/249
 112 2974 0         FOF_GList    633.7825   19494.8483  fof_fof()/fof.c/337
   7 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   7 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   7 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   7 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   7 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
   7 2973 0         FOF_PList    380.2199   18861.0161  fof_fof()/fof.c/249
   7 2974 0         FOF_GList    633.6998   19494.7160  fof_fof()/fof.c/337
   7 2975 0             Group     12.8496   19507.5656  fof_fof()/fof.c/380
   7 2976 0   DomainNodeIndex      0.3458   19507.9113  force_treeallocate()/forcetree.c/4800
   7 2977 0        Nodes_base   3737.4974   23245.4087  force_treeallocate()/forcetree.c/4803
   7 2978 0     Extnodes_base   3114.5811   26359.9899  force_treeallocate()/forcetree.c/4811
   7 2979 0          Nextnode    222.3007   26582.2906  force_treeallocate()/forcetree.c/4820
   7 2980 0            Father    221.9056   26804.1962  force_treeallocate()/forcetree.c/4827
   7 2981 0           Ngblist     63.3700   26867.5662  subfind_density()/subfind_density.c/78
   7 2982 0         Dist2list    126.7400   26994.3061  subfind_density()/subfind_density.c/79
   7 2983 0    DataIndexTable     21.8182   27016.1243  subfind_density()/subfind_density.c/87
   7 2984 0      DataNodeList     58.1818   27074.3061  subfind_density()/subfind_density.c/89
   7 2985 0              Left    126.7400   27201.0461  subfind_density()/subfind_density.c/91
 120 2973 0         FOF_PList    380.3279   18861.1242  fof_fof()/fof.c/249
 120 2974 0         FOF_GList    633.8799   19495.0041  fof_fof()/fof.c/337
 120 2975 0             Group     12.1451   19507.1491  fof_fof()/fof.c/380
 112 2975 0             Group     11.9127   19506.7610  fof_fof()/fof.c/380
 112 2976 0   DomainNodeIndex      0.3458   19507.1067  force_treeallocate()/forcetree.c/4800
   7 2986 0             Right    126.7400   27327.7860  subfind_density()/subfind_density.c/92
   7 2987 0              Todo     15.8425   27343.6285  subfind_density()/subfind_density.c/93
   7 2988 0             DM_Vx    126.7400   27470.3685  subfind_density()/subfind_density.c/95
   7 2989 0             DM_Vy    126.7400   27597.1085  subfind_density()/subfind_density.c/96
   7 2990 0             DM_Vz    126.7400   27723.8484  subfind_density()/subfind_density.c/97
   7 2991 0    	  DensDataGet    114.1146   27837.9630  subfind_density()/subfind_density.c/161
 120 2976 0   DomainNodeIndex      0.3458   19507.4949  force_treeallocate()/forcetree.c/4800
 120 2977 0        Nodes_base   3737.4974   23244.9923  force_treeallocate()/forcetree.c/4803
 112 2977 0        Nodes_base   3737.4974   23244.6041  force_treeallocate()/forcetree.c/4803
 112 2978 0     Extnodes_base   3114.5811   26359.1852  force_treeallocate()/forcetree.c/4811
 112 2979 0          Nextnode    222.3007   26581.4860  force_treeallocate()/forcetree.c/4820
   7 2992 0  	  DensDataResult     85.5859   27923.5489  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=7: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.4511 MB).
task 7: endrun called with an error level of 812


 120 2978 0     Extnodes_base   3114.5811   26359.5734  force_treeallocate()/forcetree.c/4811
 120 2979 0          Nextnode    222.3007   26581.8742  force_treeallocate()/forcetree.c/4820
 112 2980 0            Father    221.9056   26803.3916  force_treeallocate()/forcetree.c/4827
 112 2981 0           Ngblist     63.3783   26866.7698  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 7
 120 2980 0            Father    221.9056   26803.7797  force_treeallocate()/forcetree.c/4827
 120 2981 0           Ngblist     63.3880   26867.1677  subfind_density()/subfind_density.c/78
 112 2982 0         Dist2list    126.7565   26993.5263  subfind_density()/subfind_density.c/79
 112 2983 0    DataIndexTable     21.8182   27015.3445  subfind_density()/subfind_density.c/87
 112 2984 0      DataNodeList     58.1818   27073.5263  subfind_density()/subfind_density.c/89
 120 2982 0         Dist2list    126.7760   26993.9437  subfind_density()/subfind_density.c/79
 112 2985 0              Left    126.7565   27200.2828  subfind_density()/subfind_density.c/91
 120 2983 0    DataIndexTable     21.8182   27015.7619  subfind_density()/subfind_density.c/87
 112 2986 0             Right    126.7565   27327.0393  subfind_density()/subfind_density.c/92
 120 2984 0      DataNodeList     58.1818   27073.9437  subfind_density()/subfind_density.c/89
 112 2987 0              Todo     15.8446   27342.8839  subfind_density()/subfind_density.c/93
 120 2985 0              Left    126.7760   27200.7197  subfind_density()/subfind_density.c/91
 112 2988 0             DM_Vx    126.7565   27469.6404  subfind_density()/subfind_density.c/95
 120 2986 0             Right    126.7760   27327.4956  subfind_density()/subfind_density.c/92
 112 2989 0             DM_Vy    126.7565   27596.3969  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2987 0              Todo     15.8470   27343.3426  subfind_density()/subfind_density.c/93
 112 2990 0             DM_Vz    126.7565   27723.1534  subfind_density()/subfind_density.c/97
 112 2991 0    	  DensDataGet    116.5422   27839.6956  subfind_density()/subfind_density.c/161
 112 2992 0  	  DensDataResult     87.4066   27927.1022  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  18 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  18 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  18 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  18 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  18 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  18 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2988 0             DM_Vx    126.7760   27470.1186  subfind_density()/subfind_density.c/95

Task=112: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.8978 MB).
task 112: endrun called with an error level of 812


  18 2973 0         FOF_PList    380.2366   18861.0329  fof_fof()/fof.c/249
  18 2974 0         FOF_GList    633.7277   19494.7606  fof_fof()/fof.c/337
  18 2975 0             Group     13.3705   19508.1311  fof_fof()/fof.c/380
 109 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 109 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 109 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 109 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 109 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 109 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 109 2973 0         FOF_PList    380.6189   18861.4151  fof_fof()/fof.c/249
 109 2974 0         FOF_GList    634.3648   19495.7799  fof_fof()/fof.c/337
 109 2975 0             Group     12.0610   19507.8410  fof_fof()/fof.c/380
 109 2976 0   DomainNodeIndex      0.3458   19508.1867  force_treeallocate()/forcetree.c/4800
 109 2977 0        Nodes_base   3737.4974   23245.6841  force_treeallocate()/forcetree.c/4803
 109 2978 0     Extnodes_base   3114.5811   26360.2653  force_treeallocate()/forcetree.c/4811
 109 2979 0          Nextnode    222.3007   26582.5660  force_treeallocate()/forcetree.c/4820
   0 2965 0     ProcessedFlag     55.4764      57.4321  allocate_memory()/allocate.c/52
 120 2989 0             DM_Vy    126.7760   27596.8946  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 112
  18 2976 0   DomainNodeIndex      0.3458   19508.4768  force_treeallocate()/forcetree.c/4800
  18 2977 0        Nodes_base   3737.4974   23245.9742  force_treeallocate()/forcetree.c/4803
 109 2980 0            Father    221.9056   26804.4716  force_treeallocate()/forcetree.c/4827
 109 2981 0           Ngblist     63.4365   26867.9081  subfind_density()/subfind_density.c/78
 109 2982 0         Dist2list    126.8730   26994.7810  subfind_density()/subfind_density.c/79
 109 2983 0    DataIndexTable     21.8182   27016.5992  subfind_density()/subfind_density.c/87
 109 2984 0      DataNodeList     58.1818   27074.7810  subfind_density()/subfind_density.c/89
   0 2966 0  NextActiveParticle    221.9056     279.3377  allocate_memory()/allocate.c/55
 120 2990 0             DM_Vz    126.7760   27723.6706  subfind_density()/subfind_density.c/97
 127 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 127 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 127 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 127 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 127 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 127 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 127 2973 0         FOF_PList    380.2635   18861.0597  fof_fof()/fof.c/249
 127 2974 0         FOF_GList    633.7724   19494.8321  fof_fof()/fof.c/337
 127 2975 0             Group     12.4086   19507.2408  fof_fof()/fof.c/380
 127 2976 0   DomainNodeIndex      0.3458   19507.5865  force_treeallocate()/forcetree.c/4800
 127 2977 0        Nodes_base   3737.4974   23245.0839  force_treeallocate()/forcetree.c/4803
 127 2978 0     Extnodes_base   3114.5811   26359.6650  force_treeallocate()/forcetree.c/4811
 127 2979 0          Nextnode    222.3007   26581.9658  force_treeallocate()/forcetree.c/4820
 127 2980 0            Father    221.9056   26803.8714  force_treeallocate()/forcetree.c/4827
 127 2981 0           Ngblist     63.3772   26867.2486  subfind_density()/subfind_density.c/78
 127 2982 0         Dist2list    126.7545   26994.0031  subfind_density()/subfind_density.c/79
 An HDF5 error was detected, we'd better stop.
  18 2978 0     Extnodes_base   3114.5811   26360.5554  force_treeallocate()/forcetree.c/4811
  18 2979 0          Nextnode    222.3007   26582.8561  force_treeallocate()/forcetree.c/4820
  18 2980 0            Father    221.9056   26804.7617  force_treeallocate()/forcetree.c/4827
 109 2985 0              Left    126.8730   27201.6540  subfind_density()/subfind_density.c/91
 109 2986 0             Right    126.8730   27328.5269  subfind_density()/subfind_density.c/92
 109 2987 0              Todo     15.8591   27344.3860  subfind_density()/subfind_density.c/93
   0 2967 0     NextInTimeBin    221.9056     501.2433  allocate_memory()/allocate.c/58
 120 2991 0    	  DensDataGet    116.2278   27839.8984  subfind_density()/subfind_density.c/161
 127 2983 0    DataIndexTable     21.8182   27015.8213  subfind_density()/subfind_density.c/87
 127 2984 0      DataNodeList     58.1818   27074.0031  subfind_density()/subfind_density.c/89
 127 2985 0              Left    126.7545   27200.7575  subfind_density()/subfind_density.c/91
 127 2986 0             Right    126.7545   27327.5120  subfind_density()/subfind_density.c/92
HDF5-DIAG: Error detected in HDF5 (1.8.12)   18 2981 0           Ngblist     63.3728   26868.1344  subfind_density()/subfind_density.c/78
  18 2982 0         Dist2list    126.7455   26994.8800  subfind_density()/subfind_density.c/79
  18 2983 0    DataIndexTable     21.8182   27016.6982  subfind_density()/subfind_density.c/87
 109 2988 0             DM_Vx    126.8730   27471.2590  subfind_density()/subfind_density.c/95
 109 2989 0             DM_Vy    126.8730   27598.1320  subfind_density()/subfind_density.c/96
 109 2990 0             DM_Vz    126.8730   27725.0049  subfind_density()/subfind_density.c/97
   0 2968 0     PrevInTimeBin    221.9056     723.1489  allocate_memory()/allocate.c/61
 120 2992 0  	  DensDataResult     87.1709   27927.0693  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 127 2987 0              Todo     15.8443   27343.3563  subfind_density()/subfind_density.c/93
 127 2988 0             DM_Vx    126.7545   27470.1108  subfind_density()/subfind_density.c/95
 127 2989 0             DM_Vy    126.7545   27596.8653  subfind_density()/subfind_density.c/96
 127 2990 0             DM_Vz    126.7545   27723.6198  subfind_density()/subfind_density.c/97
MPI-process 112  18 2984 0      DataNodeList     58.1818   27074.8800  subfind_density()/subfind_density.c/89
  18 2985 0              Left    126.7455   27201.6255  subfind_density()/subfind_density.c/91
 109 2991 0    	  DensDataGet    114.8796   27839.8845  subfind_density()/subfind_density.c/161
 109 2992 0  	  DensDataResult     86.1597   27926.0442  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   0 2969 0                 P  17752.4469   18475.5958  allocate_memory()/allocate.c/67

Task=120: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.9307 MB).
task 120: endrun called with an error level of 812


 127 2991 0    	  DensDataGet    112.3812   27836.0010  subfind_density()/subfind_density.c/161
 127 2992 0  	  DensDataResult     84.2859   27920.2869  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 112: endrun called with an error level of 124


  18 2986 0             Right    126.7455   27328.3710  subfind_density()/subfind_density.c/92

Task=109: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.9558 MB).
task 109: endrun called with an error level of 812


   0 2972 0          TopNodes      5.1369   18480.7952  domain_allocate()/domain.c/570
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 120

Task=127: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.7131 MB).
task 127: endrun called with an error level of 812


:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 112
  18 2987 0              Todo     15.8432   27344.2142  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 109
   0 2973 0         FOF_PList    380.0490   18860.8441  fof_fof()/fof.c/249
 An HDF5 error was detected, we'd better stop.
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 127
  18 2988 0             DM_Vx    126.7455   27470.9598  subfind_density()/subfind_density.c/95
   0 2974 0         FOF_GList    633.4149   19494.2591  fof_fof()/fof.c/337
HDF5-DIAG: Error detected in HDF5 (1.8.12)   18 2989 0             DM_Vy    126.7455   27597.7053  subfind_density()/subfind_density.c/96
   0 2975 0             Group     11.5806   19505.8397  fof_fof()/fof.c/380
MPI-process 120:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  18 2990 0             DM_Vz    126.7455   27724.4509  subfind_density()/subfind_density.c/97
   0 2976 0   DomainNodeIndex      0.3458   19506.1854  force_treeallocate()/forcetree.c/4800
task 120: endrun called with an error level of 124


  94 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  94 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  94 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  94 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  94 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  94 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  94 2973 0         FOF_PList    380.5853   18861.3816  fof_fof()/fof.c/249
  94 2974 0         FOF_GList    634.3089   19495.6905  fof_fof()/fof.c/337
  94 2975 0             Group     12.1627   19507.8532  fof_fof()/fof.c/380
  18 2991 0    	  DensDataGet    116.4284   27840.8793  subfind_density()/subfind_density.c/161
   0 2977 0        Nodes_base   3737.4974   23243.6828  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 120
  94 2976 0   DomainNodeIndex      0.3458   19508.1989  force_treeallocate()/forcetree.c/4800
  94 2977 0        Nodes_base   3737.4974   23245.6963  force_treeallocate()/forcetree.c/4803
  94 2978 0     Extnodes_base   3114.5811   26360.2775  force_treeallocate()/forcetree.c/4811
  18 2992 0  	  DensDataResult     87.3213   27928.2006  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   0 2978 0     Extnodes_base   3114.5811   26358.2640  force_treeallocate()/forcetree.c/4811
  94 2979 0          Nextnode    222.3007   26582.5782  force_treeallocate()/forcetree.c/4820
  94 2980 0            Father    221.9056   26804.4838  force_treeallocate()/forcetree.c/4827
  94 2981 0           Ngblist     63.4309   26867.9147  subfind_density()/subfind_density.c/78

Task=18: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.7994 MB).
   0 2979 0          Nextnode    222.3007   26580.5647  force_treeallocate()/forcetree.c/4820
  94 2982 0         Dist2list    126.8618   26994.7765  subfind_density()/subfind_density.c/79
  94 2983 0    DataIndexTable     21.8182   27016.5946  subfind_density()/subfind_density.c/87
  94 2984 0      DataNodeList     58.1818   27074.7764  subfind_density()/subfind_density.c/89
task 18: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2980 0            Father    221.9056   26802.4703  force_treeallocate()/forcetree.c/4827
  94 2985 0              Left    126.8618   27201.6382  subfind_density()/subfind_density.c/91
  94 2986 0             Right    126.8618   27328.5000  subfind_density()/subfind_density.c/92
  94 2987 0              Todo     15.8577   27344.3577  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 18
  43 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  43 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  43 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  43 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  43 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  43 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  43 2973 0         FOF_PList    380.5844   18861.3806  fof_fof()/fof.c/249
  43 2974 0         FOF_GList    634.3073   19495.6880  fof_fof()/fof.c/337
  43 2975 0             Group     10.7173   19506.4053  fof_fof()/fof.c/380
  43 2976 0   DomainNodeIndex      0.3458   19506.7511  force_treeallocate()/forcetree.c/4800
  43 2977 0        Nodes_base   3737.4974   23244.2484  force_treeallocate()/forcetree.c/4803
  43 2978 0     Extnodes_base   3114.5811   26358.8296  force_treeallocate()/forcetree.c/4811
  43 2979 0          Nextnode    222.3007   26581.1303  force_treeallocate()/forcetree.c/4820
  43 2980 0            Father    221.9056   26803.0359  force_treeallocate()/forcetree.c/4827
  43 2981 0           Ngblist     63.4307   26866.4666  subfind_density()/subfind_density.c/78
  43 2982 0         Dist2list    126.8615   26993.3281  subfind_density()/subfind_density.c/79
   0 2981 0           Ngblist     63.3415   26865.8118  subfind_density()/subfind_density.c/78
  94 2988 0             DM_Vx    126.8618   27471.2195  subfind_density()/subfind_density.c/95
  43 2983 0    DataIndexTable     21.8182   27015.1463  subfind_density()/subfind_density.c/87
  43 2984 0      DataNodeList     58.1818   27073.3281  subfind_density()/subfind_density.c/89
  43 2985 0              Left    126.8615   27200.1895  subfind_density()/subfind_density.c/91
  43 2986 0             Right    126.8615   27327.0510  subfind_density()/subfind_density.c/92
   0 2982 0         Dist2list    126.6830   26992.4948  subfind_density()/subfind_density.c/79
  94 2989 0             DM_Vy    126.8618   27598.0813  subfind_density()/subfind_density.c/96
  43 2987 0              Todo     15.8577   27342.9087  subfind_density()/subfind_density.c/93
  43 2988 0             DM_Vx    126.8615   27469.7702  subfind_density()/subfind_density.c/95
  43 2989 0             DM_Vy    126.8615   27596.6316  subfind_density()/subfind_density.c/96
  43 2990 0             DM_Vz    126.8615   27723.4931  subfind_density()/subfind_density.c/97
   0 2983 0    DataIndexTable     21.8182   27014.3129  subfind_density()/subfind_density.c/87
  94 2990 0             DM_Vz    126.8618   27724.9430  subfind_density()/subfind_density.c/97
  43 2991 0    	  DensDataGet    112.8598   27836.3529  subfind_density()/subfind_density.c/161
  43 2992 0  	  DensDataResult     84.6449   27920.9977  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   0 2984 0      DataNodeList     58.1818   27072.4947  subfind_density()/subfind_density.c/89
  94 2991 0    	  DensDataGet    124.1689   27849.1120  subfind_density()/subfind_density.c/161

Task=43: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.0023 MB).
task 43: endrun called with an error level of 812


   0 2985 0              Left    126.6830   27199.1777  subfind_density()/subfind_density.c/91
  94 2992 0  	  DensDataResult     93.1267   27942.2387  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 43
   0 2986 0             Right    126.6830   27325.8607  subfind_density()/subfind_density.c/92

Task=94: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.7613 MB).
task 94: endrun called with an error level of 812


   0 2987 0              Todo     15.8354   27341.6961  subfind_density()/subfind_density.c/93
   0 2988 0             DM_Vx    126.6830   27468.3791  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 94
   0 2989 0             DM_Vy    126.6830   27595.0620  subfind_density()/subfind_density.c/96
   0 2990 0             DM_Vz    126.6830   27721.7450  subfind_density()/subfind_density.c/97
   0 2991 0    	  DensDataGet    124.8513   27846.5963  subfind_density()/subfind_density.c/161
   0 2992 0  	  DensDataResult     93.6385   27940.2348  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=0: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.7652 MB).
task 0: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 0
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
task 0: endrun called with an error level of 124


    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 0
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  95 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  95 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  95 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  95 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  95 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  95 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  95 2973 0         FOF_PList    380.6902   18861.4864  fof_fof()/fof.c/249
  95 2974 0         FOF_GList    634.4837   19495.9701  fof_fof()/fof.c/337
  95 2975 0             Group     11.5993   19507.5694  fof_fof()/fof.c/380
  95 2976 0   DomainNodeIndex      0.3458   19507.9152  force_treeallocate()/forcetree.c/4800
  95 2977 0        Nodes_base   3737.4974   23245.4126  force_treeallocate()/forcetree.c/4803
  95 2978 0     Extnodes_base   3114.5811   26359.9937  force_treeallocate()/forcetree.c/4811
  95 2979 0          Nextnode    222.3007   26582.2944  force_treeallocate()/forcetree.c/4820
  95 2980 0            Father    221.9056   26804.2000  force_treeallocate()/forcetree.c/4827
  95 2981 0           Ngblist     63.4484   26867.6484  subfind_density()/subfind_density.c/78
  95 2982 0         Dist2list    126.8967   26994.5451  subfind_density()/subfind_density.c/79
  95 2983 0    DataIndexTable     21.8182   27016.3633  subfind_density()/subfind_density.c/87
  95 2984 0      DataNodeList     58.1818   27074.5451  subfind_density()/subfind_density.c/89
  95 2985 0              Left    126.8967   27201.4418  subfind_density()/subfind_density.c/91
  95 2986 0             Right    126.8967   27328.3386  subfind_density()/subfind_density.c/92
  95 2987 0              Todo     15.8621   27344.2007  subfind_density()/subfind_density.c/93
  95 2988 0             DM_Vx    126.8967   27471.0974  subfind_density()/subfind_density.c/95
  95 2989 0             DM_Vy    126.8967   27597.9941  subfind_density()/subfind_density.c/96
  95 2990 0             DM_Vz    126.8967   27724.8909  subfind_density()/subfind_density.c/97
  95 2991 0    	  DensDataGet    116.3651   27841.2559  subfind_density()/subfind_density.c/161
  95 2992 0  	  DensDataResult     87.2738   27928.5297  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=95: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.4703 MB).
task 95: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 110 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 110 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 110 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 110 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 110 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 110 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 110 2973 0         FOF_PList    380.3117   18861.1079  fof_fof()/fof.c/249
 110 2974 0         FOF_GList    633.8528   19494.9607  fof_fof()/fof.c/337
 110 2975 0             Group     12.1015   19507.0622  fof_fof()/fof.c/380
 110 2976 0   DomainNodeIndex      0.3458   19507.4080  force_treeallocate()/forcetree.c/4800
 110 2977 0        Nodes_base   3737.4974   23244.9054  force_treeallocate()/forcetree.c/4803
 110 2978 0     Extnodes_base   3114.5811   26359.4865  force_treeallocate()/forcetree.c/4811
 110 2979 0          Nextnode    222.3007   26581.7872  force_treeallocate()/forcetree.c/4820
 110 2980 0            Father    221.9056   26803.6928  force_treeallocate()/forcetree.c/4827
 110 2981 0           Ngblist     63.3853   26867.0781  subfind_density()/subfind_density.c/78
 110 2982 0         Dist2list    126.7706   26993.8487  subfind_density()/subfind_density.c/79
 110 2983 0    DataIndexTable     21.8182   27015.6668  subfind_density()/subfind_density.c/87
 110 2984 0      DataNodeList     58.1818   27073.8486  subfind_density()/subfind_density.c/89
 110 2985 0              Left    126.7706   27200.6192  subfind_density()/subfind_density.c/91
 110 2986 0             Right    126.7706   27327.3898  subfind_density()/subfind_density.c/92
 110 2987 0              Todo     15.8463   27343.2361  subfind_density()/subfind_density.c/93
 110 2988 0             DM_Vx    126.7706   27470.0066  subfind_density()/subfind_density.c/95
 110 2989 0             DM_Vy    126.7706   27596.7772  subfind_density()/subfind_density.c/96
 110 2990 0             DM_Vz    126.7706   27723.5478  subfind_density()/subfind_density.c/97
 110 2991 0    	  DensDataGet    123.0773   27846.6250  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 110 2992 0  	  DensDataResult     92.3080   27938.9330  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   3 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   3 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   3 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   3 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   3 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   3 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
   3 2973 0         FOF_PList    380.5470   18861.3432  fof_fof()/fof.c/249
   3 2974 0         FOF_GList    634.2449   19495.5881  fof_fof()/fof.c/337
   3 2975 0             Group     12.8610   19508.4491  fof_fof()/fof.c/380
   3 2976 0   DomainNodeIndex      0.3458   19508.7949  force_treeallocate()/forcetree.c/4800
   3 2977 0        Nodes_base   3737.4974   23246.2923  force_treeallocate()/forcetree.c/4803
   3 2978 0     Extnodes_base   3114.5811   26360.8734  force_treeallocate()/forcetree.c/4811
   3 2979 0          Nextnode    222.3007   26583.1741  force_treeallocate()/forcetree.c/4820

Task=110: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.067 MB).
task 110: endrun called with an error level of 812


   3 2980 0            Father    221.9056   26805.0797  force_treeallocate()/forcetree.c/4827
   3 2981 0           Ngblist     63.4245   26868.5042  subfind_density()/subfind_density.c/78
   3 2982 0         Dist2list    126.8490   26995.3532  subfind_density()/subfind_density.c/79
   3 2983 0    DataIndexTable     21.8182   27017.1714  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 110
   3 2984 0      DataNodeList     58.1818   27075.3532  subfind_density()/subfind_density.c/89
   3 2985 0              Left    126.8490   27202.2022  subfind_density()/subfind_density.c/91
   3 2986 0             Right    126.8490   27329.0511  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   3 2987 0              Todo     15.8561   27344.9073  subfind_density()/subfind_density.c/93
   3 2988 0             DM_Vx    126.8490   27471.7562  subfind_density()/subfind_density.c/95
   3 2989 0             DM_Vy    126.8490   27598.6052  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 113 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 113 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 113 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 113 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 113 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 113 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 113 2973 0         FOF_PList    380.6378   18861.4340  fof_fof()/fof.c/249
 113 2974 0         FOF_GList    634.3963   19495.8304  fof_fof()/fof.c/337
 113 2975 0             Group     11.7435   19507.5739  fof_fof()/fof.c/380
 113 2976 0   DomainNodeIndex      0.3458   19507.9196  force_treeallocate()/forcetree.c/4800
 113 2977 0        Nodes_base   3737.4974   23245.4170  force_treeallocate()/forcetree.c/4803
 113 2978 0     Extnodes_base   3114.5811   26359.9982  force_treeallocate()/forcetree.c/4811
 113 2979 0          Nextnode    222.3007   26582.2989  force_treeallocate()/forcetree.c/4820
   3 2990 0             DM_Vz    126.8490   27725.4542  subfind_density()/subfind_density.c/97
   3 2991 0    	  DensDataGet    131.4233   27856.8775  subfind_density()/subfind_density.c/161
   3 2992 0  	  DensDataResult     98.5675   27955.4450  subfind_density()/subfind_density.c/201
  14 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  14 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  14 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  14 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  14 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  14 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  14 2973 0         FOF_PList    380.4175   18861.2137  fof_fof()/fof.c/249
  14 2974 0         FOF_GList    634.0292   19495.2429  fof_fof()/fof.c/337
  14 2975 0             Group     11.5910   19506.8339  fof_fof()/fof.c/380
  14 2976 0   DomainNodeIndex      0.3458   19507.1796  force_treeallocate()/forcetree.c/4800
  14 2977 0        Nodes_base   3737.4974   23244.6770  force_treeallocate()/forcetree.c/4803
 113 2980 0            Father    221.9056   26804.2045  force_treeallocate()/forcetree.c/4827
 113 2981 0           Ngblist     63.4396   26867.6441  subfind_density()/subfind_density.c/78
 113 2982 0         Dist2list    126.8793   26994.5234  subfind_density()/subfind_density.c/79
 113 2983 0    DataIndexTable     21.8182   27016.3416  subfind_density()/subfind_density.c/87
----------------------------------------------------------------------------------------
  14 2978 0     Extnodes_base   3114.5811   26359.2582  force_treeallocate()/forcetree.c/4811
  14 2979 0          Nextnode    222.3007   26581.5589  force_treeallocate()/forcetree.c/4820
  14 2980 0            Father    221.9056   26803.4645  force_treeallocate()/forcetree.c/4827
 113 2984 0      DataNodeList     58.1818   27074.5234  subfind_density()/subfind_density.c/89
 113 2985 0              Left    126.8793   27201.4026  subfind_density()/subfind_density.c/91
 113 2986 0             Right    126.8793   27328.2819  subfind_density()/subfind_density.c/92

Task=3: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=44.555 MB).
  14 2981 0           Ngblist     63.4029   26866.8674  subfind_density()/subfind_density.c/78
  14 2982 0         Dist2list    126.8058   26993.6732  subfind_density()/subfind_density.c/79
  14 2983 0    DataIndexTable     21.8182   27015.4914  subfind_density()/subfind_density.c/87
 113 2987 0              Todo     15.8599   27344.1418  subfind_density()/subfind_density.c/93
 113 2988 0             DM_Vx    126.8793   27471.0211  subfind_density()/subfind_density.c/95
 113 2989 0             DM_Vy    126.8793   27597.9003  subfind_density()/subfind_density.c/96
task 3: endrun called with an error level of 812


  14 2984 0      DataNodeList     58.1818   27073.6732  subfind_density()/subfind_density.c/89
  14 2985 0              Left    126.8058   27200.4790  subfind_density()/subfind_density.c/91
  14 2986 0             Right    126.8058   27327.2849  subfind_density()/subfind_density.c/92
 113 2990 0             DM_Vz    126.8793   27724.7796  subfind_density()/subfind_density.c/97
 113 2991 0    	  DensDataGet    121.5874   27846.3670  subfind_density()/subfind_density.c/161
 113 2992 0  	  DensDataResult     91.1906   27937.5575  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 3
  14 2987 0              Todo     15.8507   27343.1356  subfind_density()/subfind_density.c/93
  14 2988 0             DM_Vx    126.8058   27469.9414  subfind_density()/subfind_density.c/95
  14 2989 0             DM_Vy    126.8058   27596.7473  subfind_density()/subfind_density.c/96

Task=113: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.4425 MB).
task 113: endrun called with an error level of 812


  14 2990 0             DM_Vz    126.8058   27723.5531  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 113
  14 2991 0    	  DensDataGet    120.8868   27844.4399  subfind_density()/subfind_density.c/161
  14 2992 0  	  DensDataResult     90.6651   27935.1051  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=14: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.8949 MB).
task 14: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 14
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 107 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 107 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 107 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 107 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 107 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 107 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 107 2973 0         FOF_PList    380.6881   18861.4844  fof_fof()/fof.c/249
 107 2974 0         FOF_GList    634.4802   19495.9646  fof_fof()/fof.c/337
 107 2975 0             Group     11.6730   19507.6375  fof_fof()/fof.c/380
 107 2976 0   DomainNodeIndex      0.3458   19507.9833  force_treeallocate()/forcetree.c/4800
 107 2977 0        Nodes_base   3737.4974   23245.4807  force_treeallocate()/forcetree.c/4803
 107 2978 0     Extnodes_base   3114.5811   26360.0618  force_treeallocate()/forcetree.c/4811
 107 2979 0          Nextnode    222.3007   26582.3626  force_treeallocate()/forcetree.c/4820
 107 2980 0            Father    221.9056   26804.2681  force_treeallocate()/forcetree.c/4827
 107 2981 0           Ngblist     63.4480   26867.7162  subfind_density()/subfind_density.c/78
 107 2982 0         Dist2list    126.8960   26994.6122  subfind_density()/subfind_density.c/79
 107 2983 0    DataIndexTable     21.8182   27016.4304  subfind_density()/subfind_density.c/87
 107 2984 0      DataNodeList     58.1818   27074.6122  subfind_density()/subfind_density.c/89
 107 2985 0              Left    126.8960   27201.5082  subfind_density()/subfind_density.c/91
 107 2986 0             Right    126.8960   27328.4043  subfind_density()/subfind_density.c/92
 107 2987 0              Todo     15.8620   27344.2663  subfind_density()/subfind_density.c/93
 107 2988 0             DM_Vx    126.8960   27471.1623  subfind_density()/subfind_density.c/95
 107 2989 0             DM_Vy    126.8960   27598.0583  subfind_density()/subfind_density.c/96
 107 2990 0             DM_Vz    126.8960   27724.9544  subfind_density()/subfind_density.c/97
 107 2991 0    	  DensDataGet    112.2996   27837.2540  subfind_density()/subfind_density.c/161
 107 2992 0  	  DensDataResult     84.2247   27921.4786  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=107: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.5214 MB).
task 107: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 107
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   2 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   2 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   2 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   2 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   2 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   2 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
   2 2973 0         FOF_PList    380.1819   18860.9781  fof_fof()/fof.c/249
 125 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 125 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 125 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 125 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 125 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 125 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 125 2973 0         FOF_PList    380.4833   18861.2795  fof_fof()/fof.c/249
 125 2974 0         FOF_GList    634.1388   19495.4183  fof_fof()/fof.c/337
 125 2975 0             Group     13.3653   19508.7836  fof_fof()/fof.c/380
 125 2976 0   DomainNodeIndex      0.3458   19509.1294  force_treeallocate()/forcetree.c/4800
 125 2977 0        Nodes_base   3737.4974   23246.6267  force_treeallocate()/forcetree.c/4803
 125 2978 0     Extnodes_base   3114.5811   26361.2079  force_treeallocate()/forcetree.c/4811
 125 2979 0          Nextnode    222.3007   26583.5086  force_treeallocate()/forcetree.c/4820
 125 2980 0            Father    221.9056   26805.4142  force_treeallocate()/forcetree.c/4827
 125 2981 0           Ngblist     63.4139   26868.8281  subfind_density()/subfind_density.c/78
 125 2982 0         Dist2list    126.8278   26995.6558  subfind_density()/subfind_density.c/79
 125 2983 0    DataIndexTable     21.8182   27017.4740  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   2 2974 0         FOF_GList    633.6364   19494.6145  fof_fof()/fof.c/337
   2 2975 0             Group     11.6045   19506.2190  fof_fof()/fof.c/380
 125 2984 0      DataNodeList     58.1818   27075.6558  subfind_density()/subfind_density.c/89
 125 2985 0              Left    126.8278   27202.4836  subfind_density()/subfind_density.c/91
 125 2986 0             Right    126.8278   27329.3113  subfind_density()/subfind_density.c/92
 125 2987 0              Todo     15.8535   27345.1648  subfind_density()/subfind_density.c/93
 125 2988 0             DM_Vx    126.8278   27471.9926  subfind_density()/subfind_density.c/95
 125 2989 0             DM_Vy    126.8278   27598.8203  subfind_density()/subfind_density.c/96
 125 2990 0             DM_Vz    126.8278   27725.6481  subfind_density()/subfind_density.c/97
 118 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 118 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 118 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 118 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 118 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 118 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 118 2973 0         FOF_PList    380.6443   18861.4405  fof_fof()/fof.c/249
 118 2974 0         FOF_GList    634.4072   19495.8477  fof_fof()/fof.c/337
 118 2975 0             Group     12.1264   19507.9741  fof_fof()/fof.c/380
 118 2976 0   DomainNodeIndex      0.3458   19508.3198  force_treeallocate()/forcetree.c/4800
 118 2977 0        Nodes_base   3737.4974   23245.8172  force_treeallocate()/forcetree.c/4803
 118 2978 0     Extnodes_base   3114.5811   26360.3984  force_treeallocate()/forcetree.c/4811
   9 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   9 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   9 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   9 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   9 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   9 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
   9 2973 0         FOF_PList    380.6053   18861.4016  fof_fof()/fof.c/249
   9 2974 0         FOF_GList    634.3422   19495.7438  fof_fof()/fof.c/337
   9 2975 0             Group     11.9106   19507.6544  fof_fof()/fof.c/380
   9 2976 0   DomainNodeIndex      0.3458   19508.0001  force_treeallocate()/forcetree.c/4800
   9 2977 0        Nodes_base   3737.4974   23245.4975  force_treeallocate()/forcetree.c/4803
   9 2978 0     Extnodes_base   3114.5811   26360.0787  force_treeallocate()/forcetree.c/4811
   9 2979 0          Nextnode    222.3007   26582.3794  force_treeallocate()/forcetree.c/4820
   9 2980 0            Father    221.9056   26804.2850  force_treeallocate()/forcetree.c/4827
   9 2981 0           Ngblist     63.4342   26867.7192  subfind_density()/subfind_density.c/78
   9 2982 0         Dist2list    126.8684   26994.5877  subfind_density()/subfind_density.c/79
   9 2983 0    DataIndexTable     21.8182   27016.4058  subfind_density()/subfind_density.c/87
   2 2976 0   DomainNodeIndex      0.3458   19506.5648  force_treeallocate()/forcetree.c/4800
   2 2977 0        Nodes_base   3737.4974   23244.0622  force_treeallocate()/forcetree.c/4803
   2 2978 0     Extnodes_base   3114.5811   26358.6433  force_treeallocate()/forcetree.c/4811
   2 2979 0          Nextnode    222.3007   26580.9440  force_treeallocate()/forcetree.c/4820
 125 2991 0    	  DensDataGet    123.6490   27849.2971  subfind_density()/subfind_density.c/161
 125 2992 0  	  DensDataResult     92.7367   27942.0338  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=125: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.9662 MB).
 118 2979 0          Nextnode    222.3007   26582.6991  force_treeallocate()/forcetree.c/4820
 118 2980 0            Father    221.9056   26804.6047  force_treeallocate()/forcetree.c/4827
   9 2984 0      DataNodeList     58.1818   27074.5876  subfind_density()/subfind_density.c/89
   9 2985 0              Left    126.8684   27201.4561  subfind_density()/subfind_density.c/91
   9 2986 0             Right    126.8684   27328.3245  subfind_density()/subfind_density.c/92
   9 2987 0              Todo     15.8586   27344.1831  subfind_density()/subfind_density.c/93
   9 2988 0             DM_Vx    126.8684   27471.0515  subfind_density()/subfind_density.c/95
   9 2989 0             DM_Vy    126.8684   27597.9200  subfind_density()/subfind_density.c/96
   9 2990 0             DM_Vz    126.8684   27724.7884  subfind_density()/subfind_density.c/97
   2 2980 0            Father    221.9056   26802.8496  force_treeallocate()/forcetree.c/4827
   2 2981 0           Ngblist     63.3636   26866.2133  subfind_density()/subfind_density.c/78
   2 2982 0         Dist2list    126.7273   26992.9406  subfind_density()/subfind_density.c/79
task 125: endrun called with an error level of 812


 118 2981 0           Ngblist     63.4407   26868.0454  subfind_density()/subfind_density.c/78
 118 2982 0         Dist2list    126.8814   26994.9268  subfind_density()/subfind_density.c/79
 118 2983 0    DataIndexTable     21.8182   27016.7450  subfind_density()/subfind_density.c/87
   9 2991 0    	  DensDataGet    107.8904   27832.6788  subfind_density()/subfind_density.c/161
   9 2992 0  	  DensDataResult     80.9178   27913.5966  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=9: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.4034 MB).
task 9: endrun called with an error level of 812


   2 2983 0    DataIndexTable     21.8182   27014.7587  subfind_density()/subfind_density.c/87
   2 2984 0      DataNodeList     58.1818   27072.9405  subfind_density()/subfind_density.c/89
   2 2985 0              Left    126.7273   27199.6678  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 125
 118 2984 0      DataNodeList     58.1818   27074.9268  subfind_density()/subfind_density.c/89
 118 2985 0              Left    126.8814   27201.8082  subfind_density()/subfind_density.c/91
 118 2986 0             Right    126.8814   27328.6897  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 9
   2 2986 0             Right    126.7273   27326.3951  subfind_density()/subfind_density.c/92
   2 2987 0              Todo     15.8409   27342.2360  subfind_density()/subfind_density.c/93
   2 2988 0             DM_Vx    126.7273   27468.9633  subfind_density()/subfind_density.c/95
 118 2987 0              Todo     15.8602   27344.5499  subfind_density()/subfind_density.c/93
 118 2988 0             DM_Vx    126.8814   27471.4313  subfind_density()/subfind_density.c/95
 118 2989 0             DM_Vy    126.8814   27598.3127  subfind_density()/subfind_density.c/96
   2 2989 0             DM_Vy    126.7273   27595.6906  subfind_density()/subfind_density.c/96
 118 2990 0             DM_Vz    126.8814   27725.1942  subfind_density()/subfind_density.c/97
   2 2990 0             DM_Vz    126.7273   27722.4179  subfind_density()/subfind_density.c/97
 118 2991 0    	  DensDataGet    141.4137   27866.6078  subfind_density()/subfind_density.c/161
   2 2991 0    	  DensDataGet    121.3486   27843.7665  subfind_density()/subfind_density.c/161
 118 2992 0  	  DensDataResult    106.0603   27972.6681  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   2 2992 0  	  DensDataResult     91.0115   27934.7780  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=118: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=27.3319 MB).
task 118: endrun called with an error level of 812



Task=2: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.222 MB).
task 2: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 118
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 2
 126 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 126 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 126 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 126 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 126 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 126 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 126 2973 0         FOF_PList    380.6676   18861.4639  fof_fof()/fof.c/249
 126 2974 0         FOF_GList    634.4461   19495.9099  fof_fof()/fof.c/337
 126 2975 0             Group     12.9461   19508.8560  fof_fof()/fof.c/380
 126 2976 0   DomainNodeIndex      0.3458   19509.2018  force_treeallocate()/forcetree.c/4800
 126 2977 0        Nodes_base   3737.4974   23246.6992  force_treeallocate()/forcetree.c/4803
 126 2978 0     Extnodes_base   3114.5811   26361.2803  force_treeallocate()/forcetree.c/4811
 126 2979 0          Nextnode    222.3007   26583.5811  force_treeallocate()/forcetree.c/4820
 126 2980 0            Father    221.9056   26805.4866  force_treeallocate()/forcetree.c/4827
 126 2981 0           Ngblist     63.4446   26868.9313  subfind_density()/subfind_density.c/78
 126 2982 0         Dist2list    126.8892   26995.8205  subfind_density()/subfind_density.c/79
 126 2983 0    DataIndexTable     21.8182   27017.6386  subfind_density()/subfind_density.c/87
 126 2984 0      DataNodeList     58.1818   27075.8204  subfind_density()/subfind_density.c/89
 126 2985 0              Left    126.8892   27202.7097  subfind_density()/subfind_density.c/91
 126 2986 0             Right    126.8892   27329.5989  subfind_density()/subfind_density.c/92
 126 2987 0              Todo     15.8612   27345.4600  subfind_density()/subfind_density.c/93
 126 2988 0             DM_Vx    126.8892   27472.3492  subfind_density()/subfind_density.c/95
 126 2989 0             DM_Vy    126.8892   27599.2384  subfind_density()/subfind_density.c/96
 126 2990 0             DM_Vz    126.8892   27726.1277  subfind_density()/subfind_density.c/97
 126 2991 0    	  DensDataGet    115.8178   27841.9455  subfind_density()/subfind_density.c/161
 126 2992 0  	  DensDataResult     86.8634   27928.8088  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=126: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.1912 MB).
task 126: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 126
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 115 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 115 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 115 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 115 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 115 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 115 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 115 2973 0         FOF_PList    380.0888   18860.8850  fof_fof()/fof.c/249
 115 2974 0         FOF_GList    633.4813   19494.3662  fof_fof()/fof.c/337
 115 2975 0             Group     13.2408   19507.6070  fof_fof()/fof.c/380
 115 2976 0   DomainNodeIndex      0.3458   19507.9528  force_treeallocate()/forcetree.c/4800
 115 2977 0        Nodes_base   3737.4974   23245.4502  force_treeallocate()/forcetree.c/4803
 115 2978 0     Extnodes_base   3114.5811   26360.0313  force_treeallocate()/forcetree.c/4811
 115 2979 0          Nextnode    222.3007   26582.3320  force_treeallocate()/forcetree.c/4820
 115 2980 0            Father    221.9056   26804.2376  force_treeallocate()/forcetree.c/4827
 115 2981 0           Ngblist     63.3481   26867.5858  subfind_density()/subfind_density.c/78
 115 2982 0         Dist2list    126.6963   26994.2820  subfind_density()/subfind_density.c/79
 115 2983 0    DataIndexTable     21.8182   27016.1002  subfind_density()/subfind_density.c/87
 115 2984 0      DataNodeList     58.1818   27074.2820  subfind_density()/subfind_density.c/89
 115 2985 0              Left    126.6963   27200.9782  subfind_density()/subfind_density.c/91
 115 2986 0             Right    126.6963   27327.6745  subfind_density()/subfind_density.c/92
 115 2987 0              Todo     15.8370   27343.5115  subfind_density()/subfind_density.c/93
 115 2988 0             DM_Vx    126.6963   27470.2078  subfind_density()/subfind_density.c/95
 115 2989 0             DM_Vy    126.6963   27596.9040  subfind_density()/subfind_density.c/96
 115 2990 0             DM_Vz    126.6963   27723.6003  subfind_density()/subfind_density.c/97
 115 2991 0    	  DensDataGet    127.6661   27851.2663  subfind_density()/subfind_density.c/161
 115 2992 0  	  DensDataResult     95.7496   27947.0159  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=115: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.9841 MB).
task 115: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 115
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 106 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 106 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 106 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 106 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 106 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  21 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  21 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  21 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  21 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  21 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  21 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  21 2973 0         FOF_PList    380.5751   18861.3713  fof_fof()/fof.c/249
  21 2974 0         FOF_GList    634.2918   19495.6631  fof_fof()/fof.c/337
  21 2975 0             Group     12.3381   19508.0011  fof_fof()/fof.c/380
  21 2976 0   DomainNodeIndex      0.3458   19508.3469  force_treeallocate()/forcetree.c/4800
  21 2977 0        Nodes_base   3737.4974   23245.8443  force_treeallocate()/forcetree.c/4803
  21 2978 0     Extnodes_base   3114.5811   26360.4254  force_treeallocate()/forcetree.c/4811
  21 2979 0          Nextnode    222.3007   26582.7261  force_treeallocate()/forcetree.c/4820
  21 2980 0            Father    221.9056   26804.6317  force_treeallocate()/forcetree.c/4827
  21 2981 0           Ngblist     63.4292   26868.0609  subfind_density()/subfind_density.c/78
  21 2982 0         Dist2list    126.8584   26994.9193  subfind_density()/subfind_density.c/79
  21 2983 0    DataIndexTable     21.8182   27016.7374  subfind_density()/subfind_density.c/87
 106 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 106 2973 0         FOF_PList    380.3545   18861.1508  fof_fof()/fof.c/249
 106 2974 0         FOF_GList    633.9242   19495.0750  fof_fof()/fof.c/337
  21 2984 0      DataNodeList     58.1818   27074.9192  subfind_density()/subfind_density.c/89
  21 2985 0              Left    126.8584   27201.7776  subfind_density()/subfind_density.c/91
  21 2986 0             Right    126.8584   27328.6359  subfind_density()/subfind_density.c/92
  21 2987 0              Todo     15.8573   27344.4932  subfind_density()/subfind_density.c/93
  21 2988 0             DM_Vx    126.8584   27471.3516  subfind_density()/subfind_density.c/95
  21 2989 0             DM_Vy    126.8584   27598.2099  subfind_density()/subfind_density.c/96
 106 2975 0             Group     12.5881   19507.6631  fof_fof()/fof.c/380
 106 2976 0   DomainNodeIndex      0.3458   19508.0089  force_treeallocate()/forcetree.c/4800
 106 2977 0        Nodes_base   3737.4974   23245.5062  force_treeallocate()/forcetree.c/4803
  21 2990 0             DM_Vz    126.8584   27725.0683  subfind_density()/subfind_density.c/97
  21 2991 0    	  DensDataGet    128.0315   27853.0998  subfind_density()/subfind_density.c/161
  21 2992 0  	  DensDataResult     96.0236   27949.1234  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 106 2978 0     Extnodes_base   3114.5811   26360.0874  force_treeallocate()/forcetree.c/4811
 106 2979 0          Nextnode    222.3007   26582.3881  force_treeallocate()/forcetree.c/4820
 106 2980 0            Father    221.9056   26804.2937  force_treeallocate()/forcetree.c/4827

Task=21: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.8766 MB).
task 21: endrun called with an error level of 812


 106 2981 0           Ngblist     63.3924   26867.6861  subfind_density()/subfind_density.c/78
 106 2982 0         Dist2list    126.7848   26994.4710  subfind_density()/subfind_density.c/79
 106 2983 0    DataIndexTable     21.8182   27016.2892  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 21
 106 2984 0      DataNodeList     58.1818   27074.4710  subfind_density()/subfind_density.c/89
 106 2985 0              Left    126.7848   27201.2558  subfind_density()/subfind_density.c/91
 106 2986 0             Right    126.7848   27328.0406  subfind_density()/subfind_density.c/92
 106 2987 0              Todo     15.8481   27343.8887  subfind_density()/subfind_density.c/93
 106 2988 0             DM_Vx    126.7848   27470.6736  subfind_density()/subfind_density.c/95
 106 2989 0             DM_Vy    126.7848   27597.4584  subfind_density()/subfind_density.c/96
 106 2990 0             DM_Vz    126.7848   27724.2433  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 106 2991 0    	  DensDataGet    122.0914   27846.3347  subfind_density()/subfind_density.c/161
  92 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  92 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  92 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  92 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  92 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  92 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  92 2973 0         FOF_PList    380.3119   18861.1081  fof_fof()/fof.c/249
  92 2974 0         FOF_GList    633.8532   19494.9613  fof_fof()/fof.c/337
  92 2975 0             Group     11.6263   19506.5876  fof_fof()/fof.c/380
 106 2992 0  	  DensDataResult     91.5686   27937.9033  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  92 2976 0   DomainNodeIndex      0.3458   19506.9334  force_treeallocate()/forcetree.c/4800
  92 2977 0        Nodes_base   3737.4974   23244.4307  force_treeallocate()/forcetree.c/4803

  92 2978 0     Extnodes_base   3114.5811   26359.0119  force_treeallocate()/forcetree.c/4811
  92 2979 0          Nextnode    222.3007   26581.3126  force_treeallocate()/forcetree.c/4820
  92 2980 0            Father    221.9056   26803.2182  force_treeallocate()/forcetree.c/4827

Task=106: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.0967 MB).
task 106: endrun called with an error level of 812


  92 2981 0           Ngblist     63.3853   26866.6035  subfind_density()/subfind_density.c/78
  92 2982 0         Dist2list    126.7706   26993.3742  subfind_density()/subfind_density.c/79
  92 2983 0    DataIndexTable     21.8182   27015.1924  subfind_density()/subfind_density.c/87
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 106
  92 2984 0      DataNodeList     58.1818   27073.3741  subfind_density()/subfind_density.c/89
  92 2985 0              Left    126.7706   27200.1448  subfind_density()/subfind_density.c/91
  92 2986 0             Right    126.7706   27326.9154  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  92 2987 0              Todo     15.8463   27342.7618  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  22 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  22 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  22 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  22 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  22 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  22 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  22 2973 0         FOF_PList    380.4218   18861.2180  fof_fof()/fof.c/249
  22 2974 0         FOF_GList    634.0363   19495.2542  fof_fof()/fof.c/337
  92 2988 0             DM_Vx    126.7706   27469.5324  subfind_density()/subfind_density.c/95
 105 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 105 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 105 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 105 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 105 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 105 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 105 2973 0         FOF_PList    379.8083   18860.6045  fof_fof()/fof.c/249
 105 2974 0         FOF_GList    633.0138   19493.6184  fof_fof()/fof.c/337
 105 2975 0             Group     13.1640   19506.7824  fof_fof()/fof.c/380
 105 2976 0   DomainNodeIndex      0.3458   19507.1281  force_treeallocate()/forcetree.c/4800
 105 2977 0        Nodes_base   3737.4974   23244.6255  force_treeallocate()/forcetree.c/4803
 105 2978 0     Extnodes_base   3114.5811   26359.2067  force_treeallocate()/forcetree.c/4811
 105 2979 0          Nextnode    222.3007   26581.5074  force_treeallocate()/forcetree.c/4820
 105 2980 0            Father    221.9056   26803.4130  force_treeallocate()/forcetree.c/4827
 105 2981 0           Ngblist     63.3014   26866.7144  subfind_density()/subfind_density.c/78
 105 2982 0         Dist2list    126.6028   26993.3171  subfind_density()/subfind_density.c/79
  22 2975 0             Group     11.1158   19506.3700  fof_fof()/fof.c/380
  22 2976 0   DomainNodeIndex      0.3458   19506.7158  force_treeallocate()/forcetree.c/4800
  92 2989 0             DM_Vy    126.7706   27596.3030  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 105 2983 0    DataIndexTable     21.8182   27015.1353  subfind_density()/subfind_density.c/87
 105 2984 0      DataNodeList     58.1818   27073.3171  subfind_density()/subfind_density.c/89
 105 2985 0              Left    126.6028   27199.9199  subfind_density()/subfind_density.c/91
 105 2986 0             Right    126.6028   27326.5226  subfind_density()/subfind_density.c/92
 105 2987 0              Todo     15.8253   27342.3480  subfind_density()/subfind_density.c/93
  22 2977 0        Nodes_base   3737.4974   23244.2132  force_treeallocate()/forcetree.c/4803
  22 2978 0     Extnodes_base   3114.5811   26358.7943  force_treeallocate()/forcetree.c/4811
  92 2990 0             DM_Vz    126.7706   27723.0737  subfind_density()/subfind_density.c/97
  13 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  13 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  13 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  13 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  13 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  13 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  13 2973 0         FOF_PList    380.0847   18860.8809  fof_fof()/fof.c/249
  13 2974 0         FOF_GList    633.4745   19494.3554  fof_fof()/fof.c/337
  13 2975 0             Group     10.9186   19505.2740  fof_fof()/fof.c/380
  13 2976 0   DomainNodeIndex      0.3458   19505.6198  force_treeallocate()/forcetree.c/4800
  13 2977 0        Nodes_base   3737.4974   23243.1171  force_treeallocate()/forcetree.c/4803
  13 2978 0     Extnodes_base   3114.5811   26357.6983  force_treeallocate()/forcetree.c/4811
  13 2979 0          Nextnode    222.3007   26579.9990  force_treeallocate()/forcetree.c/4820
  13 2980 0            Father    221.9056   26801.9046  force_treeallocate()/forcetree.c/4827
  13 2981 0           Ngblist     63.3475   26865.2521  subfind_density()/subfind_density.c/78
  13 2982 0         Dist2list    126.6949   26991.9470  subfind_density()/subfind_density.c/79
  13 2983 0    DataIndexTable     21.8182   27013.7651  subfind_density()/subfind_density.c/87
 114 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 114 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 114 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 114 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 114 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 105 2988 0             DM_Vx    126.6028   27468.9508  subfind_density()/subfind_density.c/95
 105 2989 0             DM_Vy    126.6028   27595.5535  subfind_density()/subfind_density.c/96
 105 2990 0             DM_Vz    126.6028   27722.1563  subfind_density()/subfind_density.c/97
 105 2991 0    	  DensDataGet    137.9529   27860.1092  subfind_density()/subfind_density.c/161
  22 2979 0          Nextnode    222.3007   26581.0950  force_treeallocate()/forcetree.c/4820
  22 2980 0            Father    221.9056   26803.0006  force_treeallocate()/forcetree.c/4827
  22 2981 0           Ngblist     63.4036   26866.4043  subfind_density()/subfind_density.c/78
  92 2991 0    	  DensDataGet    114.2915   27837.3652  subfind_density()/subfind_density.c/161
  13 2984 0      DataNodeList     58.1818   27071.9469  subfind_density()/subfind_density.c/89
  13 2985 0              Left    126.6949   27198.6418  subfind_density()/subfind_density.c/91
  13 2986 0             Right    126.6949   27325.3367  subfind_density()/subfind_density.c/92
  13 2987 0              Todo     15.8369   27341.1736  subfind_density()/subfind_density.c/93
  13 2988 0             DM_Vx    126.6949   27467.8685  subfind_density()/subfind_density.c/95
  13 2989 0             DM_Vy    126.6949   27594.5634  subfind_density()/subfind_density.c/96
  13 2990 0             DM_Vz    126.6949   27721.2583  subfind_density()/subfind_density.c/97
 114 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 114 2973 0         FOF_PList    380.3221   18861.1183  fof_fof()/fof.c/249
 114 2974 0         FOF_GList    633.8702   19494.9885  fof_fof()/fof.c/337
 105 2992 0  	  DensDataResult    103.4647   27963.5738  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=105: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=36.4262 MB).
  22 2982 0         Dist2list    126.8073   26993.2115  subfind_density()/subfind_density.c/79
  22 2983 0    DataIndexTable     21.8182   27015.0297  subfind_density()/subfind_density.c/87
  22 2984 0      DataNodeList     58.1818   27073.2115  subfind_density()/subfind_density.c/89
  92 2992 0  	  DensDataResult     85.7186   27923.0838  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  13 2991 0    	  DensDataGet    117.5372   27838.7955  subfind_density()/subfind_density.c/161
  13 2992 0  	  DensDataResult     88.1529   27926.9484  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=13: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.0516 MB).
 114 2975 0             Group     12.0164   19507.0049  fof_fof()/fof.c/380
 114 2976 0   DomainNodeIndex      0.3458   19507.3507  force_treeallocate()/forcetree.c/4800
 114 2977 0        Nodes_base   3737.4974   23244.8480  force_treeallocate()/forcetree.c/4803
task 105: endrun called with an error level of 812


  22 2985 0              Left    126.8073   27200.0187  subfind_density()/subfind_density.c/91

Task=92: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.9162 MB).
task 13: endrun called with an error level of 812


 114 2978 0     Extnodes_base   3114.5811   26359.4292  force_treeallocate()/forcetree.c/4811
 114 2979 0          Nextnode    222.3007   26581.7299  force_treeallocate()/forcetree.c/4820
 114 2980 0            Father    221.9056   26803.6355  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 105
  22 2986 0             Right    126.8073   27326.8260  subfind_density()/subfind_density.c/92
task 92: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 13
 114 2981 0           Ngblist     63.3870   26867.0225  subfind_density()/subfind_density.c/78
 114 2982 0         Dist2list    126.7740   26993.7966  subfind_density()/subfind_density.c/79
 114 2983 0    DataIndexTable     21.8182   27015.6147  subfind_density()/subfind_density.c/87
  22 2987 0              Todo     15.8509   27342.6769  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 92
 114 2984 0      DataNodeList     58.1818   27073.7965  subfind_density()/subfind_density.c/89
 114 2985 0              Left    126.7740   27200.5706  subfind_density()/subfind_density.c/91
  22 2988 0             DM_Vx    126.8073   27469.4841  subfind_density()/subfind_density.c/95
 114 2986 0             Right    126.7740   27327.3446  subfind_density()/subfind_density.c/92
  22 2989 0             DM_Vy    126.8073   27596.2914  subfind_density()/subfind_density.c/96
 114 2987 0              Todo     15.8468   27343.1914  subfind_density()/subfind_density.c/93
  22 2990 0             DM_Vz    126.8073   27723.0986  subfind_density()/subfind_density.c/97
 114 2988 0             DM_Vx    126.7740   27469.9654  subfind_density()/subfind_density.c/95
  22 2991 0    	  DensDataGet    111.1786   27834.2773  subfind_density()/subfind_density.c/161
 114 2989 0             DM_Vy    126.7740   27596.7394  subfind_density()/subfind_density.c/96
  22 2992 0  	  DensDataResult     83.3840   27917.6613  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 114 2990 0             DM_Vz    126.7740   27723.5135  subfind_density()/subfind_density.c/97

Task=22: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.3387 MB).
 114 2991 0    	  DensDataGet    109.6479   27833.1613  subfind_density()/subfind_density.c/161
task 22: endrun called with an error level of 812


 114 2992 0  	  DensDataResult     82.2359   27915.3973  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 22

Task=114: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.6027 MB).
task 114: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 114
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  16 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  16 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  16 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  16 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  16 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  16 2973 0         FOF_PList    380.6328   18861.4290  fof_fof()/fof.c/249
  16 2974 0         FOF_GList    634.3880   19495.8170  fof_fof()/fof.c/337
  16 2975 0             Group     11.0805   19506.8975  fof_fof()/fof.c/380
  16 2976 0   DomainNodeIndex      0.3458   19507.2432  force_treeallocate()/forcetree.c/4800
  16 2977 0        Nodes_base   3737.4974   23244.7406  force_treeallocate()/forcetree.c/4803
  16 2978 0     Extnodes_base   3114.5811   26359.3218  force_treeallocate()/forcetree.c/4811
 111 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 111 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 111 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 111 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 111 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 111 2973 0         FOF_PList    379.9251   18860.7213  fof_fof()/fof.c/249
 111 2974 0         FOF_GList    633.2085   19493.9298  fof_fof()/fof.c/337
 111 2975 0             Group     12.1015   19506.0313  fof_fof()/fof.c/380
 111 2976 0   DomainNodeIndex      0.3458   19506.3771  force_treeallocate()/forcetree.c/4800
 111 2977 0        Nodes_base   3737.4974   23243.8745  force_treeallocate()/forcetree.c/4803
 111 2978 0     Extnodes_base   3114.5811   26358.4556  force_treeallocate()/forcetree.c/4811
 111 2979 0          Nextnode    222.3007   26580.7564  force_treeallocate()/forcetree.c/4820
 111 2980 0            Father    221.9056   26802.6619  force_treeallocate()/forcetree.c/4827
 111 2981 0           Ngblist     63.3209   26865.9828  subfind_density()/subfind_density.c/78
  16 2979 0          Nextnode    222.3007   26581.6225  force_treeallocate()/forcetree.c/4820
  16 2980 0            Father    221.9056   26803.5281  force_treeallocate()/forcetree.c/4827
 111 2982 0         Dist2list    126.6417   26992.6245  subfind_density()/subfind_density.c/79
 111 2983 0    DataIndexTable     21.8182   27014.4427  subfind_density()/subfind_density.c/87
 111 2984 0      DataNodeList     58.1818   27072.6245  subfind_density()/subfind_density.c/89
 111 2985 0              Left    126.6417   27199.2662  subfind_density()/subfind_density.c/91
  16 2981 0           Ngblist     63.4388   26866.9669  subfind_density()/subfind_density.c/78
  16 2982 0         Dist2list    126.8776   26993.8445  subfind_density()/subfind_density.c/79
  16 2983 0    DataIndexTable     21.8182   27015.6627  subfind_density()/subfind_density.c/87
  16 2984 0      DataNodeList     58.1818   27073.8445  subfind_density()/subfind_density.c/89
  16 2985 0              Left    126.8776   27200.7220  subfind_density()/subfind_density.c/91
 111 2986 0             Right    126.6417   27325.9079  subfind_density()/subfind_density.c/92
 111 2987 0              Todo     15.8302   27341.7381  subfind_density()/subfind_density.c/93
 111 2988 0             DM_Vx    126.6417   27468.3798  subfind_density()/subfind_density.c/95
 111 2989 0             DM_Vy    126.6417   27595.0215  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2986 0             Right    126.8776   27327.5996  subfind_density()/subfind_density.c/92
  16 2987 0              Todo     15.8597   27343.4593  subfind_density()/subfind_density.c/93
  16 2988 0             DM_Vx    126.8776   27470.3369  subfind_density()/subfind_density.c/95
 111 2990 0             DM_Vz    126.6417   27721.6632  subfind_density()/subfind_density.c/97
 111 2991 0    	  DensDataGet    118.2012   27839.8644  subfind_density()/subfind_density.c/161
 111 2992 0  	  DensDataResult     88.6509   27928.5152  subfind_density()/subfind_density.c/201
  38 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  38 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  38 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  38 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  38 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  38 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  38 2973 0         FOF_PList    380.6191   18861.4153  fof_fof()/fof.c/249
  38 2974 0         FOF_GList    634.3652   19495.7805  fof_fof()/fof.c/337
  38 2975 0             Group     11.4572   19507.2376  fof_fof()/fof.c/380
  38 2976 0   DomainNodeIndex      0.3458   19507.5834  force_treeallocate()/forcetree.c/4800
  38 2977 0        Nodes_base   3737.4974   23245.0808  force_treeallocate()/forcetree.c/4803
  38 2978 0     Extnodes_base   3114.5811   26359.6619  force_treeallocate()/forcetree.c/4811
  16 2989 0             DM_Vy    126.8776   27597.2145  subfind_density()/subfind_density.c/96
  16 2990 0             DM_Vz    126.8776   27724.0921  subfind_density()/subfind_density.c/97
  16 2991 0    	  DensDataGet    122.1099   27846.2020  subfind_density()/subfind_density.c/161
----------------------------------------------------------------------------------------

Task=111: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.4848 MB).
task 111: endrun called with an error level of 812


  38 2979 0          Nextnode    222.3007   26581.9627  force_treeallocate()/forcetree.c/4820
  38 2980 0            Father    221.9056   26803.8682  force_treeallocate()/forcetree.c/4827
  38 2981 0           Ngblist     63.4365   26867.3048  subfind_density()/subfind_density.c/78
  16 2992 0  	  DensDataResult     91.5824   27937.7845  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 111
  38 2982 0         Dist2list    126.8730   26994.1778  subfind_density()/subfind_density.c/79
  38 2983 0    DataIndexTable     21.8182   27015.9960  subfind_density()/subfind_density.c/87
  38 2984 0      DataNodeList     58.1818   27074.1778  subfind_density()/subfind_density.c/89

Task=16: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.2155 MB).
task 16: endrun called with an error level of 812


  38 2985 0              Left    126.8730   27201.0508  subfind_density()/subfind_density.c/91
  38 2986 0             Right    126.8730   27327.9238  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 16
  38 2987 0              Todo     15.8591   27343.7830  subfind_density()/subfind_density.c/93
  38 2988 0             DM_Vx    126.8730   27470.6560  subfind_density()/subfind_density.c/95
  38 2989 0             DM_Vy    126.8730   27597.5290  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
task 16: endrun called with an error level of 124


  38 2990 0             DM_Vz    126.8730   27724.4020  subfind_density()/subfind_density.c/97
  38 2991 0    	  DensDataGet    119.4103   27843.8124  subfind_density()/subfind_density.c/161
  93 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  93 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  93 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  93 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  93 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  93 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  93 2973 0         FOF_PList    380.4551   18861.2513  fof_fof()/fof.c/249
  93 2974 0         FOF_GList    634.0918   19495.3432  fof_fof()/fof.c/337
  93 2975 0             Group     13.0084   19508.3515  fof_fof()/fof.c/380
  93 2976 0   DomainNodeIndex      0.3458   19508.6973  force_treeallocate()/forcetree.c/4800
  93 2977 0        Nodes_base   3737.4974   23246.1947  force_treeallocate()/forcetree.c/4803
  34 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  34 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  34 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  34 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  34 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  34 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  34 2973 0         FOF_PList    380.3832   18861.1794  fof_fof()/fof.c/249
  34 2974 0         FOF_GList    633.9720   19495.1514  fof_fof()/fof.c/337
  34 2975 0             Group     10.5897   19505.7411  fof_fof()/fof.c/380
  34 2976 0   DomainNodeIndex      0.3458   19506.0869  force_treeallocate()/forcetree.c/4800
  34 2977 0        Nodes_base   3737.4974   23243.5843  force_treeallocate()/forcetree.c/4803
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 16:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 16
  38 2992 0  	  DensDataResult     89.5578   27933.3701  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=38: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.6299 MB).
task 38: endrun called with an error level of 812


  93 2978 0     Extnodes_base   3114.5811   26360.7758  force_treeallocate()/forcetree.c/4811
  93 2979 0          Nextnode    222.3007   26583.0765  force_treeallocate()/forcetree.c/4820
  93 2980 0            Father    221.9056   26804.9821  force_treeallocate()/forcetree.c/4827
  93 2981 0           Ngblist     63.4092   26868.3913  subfind_density()/subfind_density.c/78
  34 2978 0     Extnodes_base   3114.5811   26358.1654  force_treeallocate()/forcetree.c/4811
  34 2979 0          Nextnode    222.3007   26580.4661  force_treeallocate()/forcetree.c/4820
  34 2980 0            Father    221.9056   26802.3717  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 38
  93 2982 0         Dist2list    126.8184   26995.2097  subfind_density()/subfind_density.c/79
  93 2983 0    DataIndexTable     21.8182   27017.0279  subfind_density()/subfind_density.c/87
  93 2984 0      DataNodeList     58.1818   27075.2097  subfind_density()/subfind_density.c/89
  34 2981 0           Ngblist     63.3972   26865.7689  subfind_density()/subfind_density.c/78
  34 2982 0         Dist2list    126.7944   26992.5633  subfind_density()/subfind_density.c/79
  34 2983 0    DataIndexTable     21.8182   27014.3815  subfind_density()/subfind_density.c/87
  93 2985 0              Left    126.8184   27202.0280  subfind_density()/subfind_density.c/91
  93 2986 0             Right    126.8184   27328.8464  subfind_density()/subfind_density.c/92
  93 2987 0              Todo     15.8523   27344.6987  subfind_density()/subfind_density.c/93
  34 2984 0      DataNodeList     58.1818   27072.5633  subfind_density()/subfind_density.c/89
  34 2985 0              Left    126.7944   27199.3577  subfind_density()/subfind_density.c/91
  34 2986 0             Right    126.7944   27326.1521  subfind_density()/subfind_density.c/92
  93 2988 0             DM_Vx    126.8184   27471.5171  subfind_density()/subfind_density.c/95
  93 2989 0             DM_Vy    126.8184   27598.3354  subfind_density()/subfind_density.c/96
  93 2990 0             DM_Vz    126.8184   27725.1538  subfind_density()/subfind_density.c/97
  34 2987 0              Todo     15.8493   27342.0014  subfind_density()/subfind_density.c/93
  34 2988 0             DM_Vx    126.7944   27468.7958  subfind_density()/subfind_density.c/95
  34 2989 0             DM_Vy    126.7944   27595.5902  subfind_density()/subfind_density.c/96
  93 2991 0    	  DensDataGet    117.9833   27843.1371  subfind_density()/subfind_density.c/161
  93 2992 0  	  DensDataResult     88.4875   27931.6246  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  34 2990 0             DM_Vz    126.7944   27722.3846  subfind_density()/subfind_density.c/97

Task=93: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.3754 MB).
task 93: endrun called with an error level of 812


  34 2991 0    	  DensDataGet    120.6603   27843.0449  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 93
  34 2992 0  	  DensDataResult     90.4953   27933.5402  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=34: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.4598 MB).
task 34: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 34
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  88 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  88 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  88 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  88 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  88 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  88 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  88 2973 0         FOF_PList    380.6809   18861.4771  fof_fof()/fof.c/249
  88 2974 0         FOF_GList    634.4681   19495.9452  fof_fof()/fof.c/337
  88 2975 0             Group     12.3578   19508.3030  fof_fof()/fof.c/380
 119 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 119 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 119 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 119 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 119 2973 0         FOF_PList    380.6812   18861.4774  fof_fof()/fof.c/249
 119 2974 0         FOF_GList    634.4687   19495.9461  fof_fof()/fof.c/337
 119 2975 0             Group     11.4717   19507.4178  fof_fof()/fof.c/380
 119 2976 0   DomainNodeIndex      0.3458   19507.7635  force_treeallocate()/forcetree.c/4800
 119 2977 0        Nodes_base   3737.4974   23245.2609  force_treeallocate()/forcetree.c/4803
 119 2978 0     Extnodes_base   3114.5811   26359.8420  force_treeallocate()/forcetree.c/4811
   8 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   8 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   8 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   8 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   8 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  88 2976 0   DomainNodeIndex      0.3458   19508.6488  force_treeallocate()/forcetree.c/4800
  88 2977 0        Nodes_base   3737.4974   23246.1461  force_treeallocate()/forcetree.c/4803
 119 2979 0          Nextnode    222.3007   26582.1428  force_treeallocate()/forcetree.c/4820
 119 2980 0            Father    221.9056   26804.0484  force_treeallocate()/forcetree.c/4827
 119 2981 0           Ngblist     63.4469   26867.4952  subfind_density()/subfind_density.c/78
 119 2982 0         Dist2list    126.8937   26994.3890  subfind_density()/subfind_density.c/79
   8 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
   8 2973 0         FOF_PList    380.4381   18861.2343  fof_fof()/fof.c/249
   8 2974 0         FOF_GList    634.0635   19495.2978  fof_fof()/fof.c/337
  88 2978 0     Extnodes_base   3114.5811   26360.7273  force_treeallocate()/forcetree.c/4811
  88 2979 0          Nextnode    222.3007   26583.0280  force_treeallocate()/forcetree.c/4820
  88 2980 0            Father    221.9056   26804.9336  force_treeallocate()/forcetree.c/4827
 119 2983 0    DataIndexTable     21.8182   27016.2071  subfind_density()/subfind_density.c/87
 119 2984 0      DataNodeList     58.1818   27074.3889  subfind_density()/subfind_density.c/89
 119 2985 0              Left    126.8937   27201.2827  subfind_density()/subfind_density.c/91
 119 2986 0             Right    126.8937   27328.1764  subfind_density()/subfind_density.c/92
   8 2975 0             Group     12.5435   19507.8413  fof_fof()/fof.c/380
   8 2976 0   DomainNodeIndex      0.3458   19508.1871  force_treeallocate()/forcetree.c/4800
   8 2977 0        Nodes_base   3737.4974   23245.6845  force_treeallocate()/forcetree.c/4803
  88 2981 0           Ngblist     63.4468   26868.3804  subfind_density()/subfind_density.c/78
  88 2982 0         Dist2list    126.8936   26995.2740  subfind_density()/subfind_density.c/79
  88 2983 0    DataIndexTable     21.8182   27017.0922  subfind_density()/subfind_density.c/87
 119 2987 0              Todo     15.8617   27344.0381  subfind_density()/subfind_density.c/93
 119 2988 0             DM_Vx    126.8937   27470.9318  subfind_density()/subfind_density.c/95
 119 2989 0             DM_Vy    126.8937   27597.8256  subfind_density()/subfind_density.c/96
   8 2978 0     Extnodes_base   3114.5811   26360.2656  force_treeallocate()/forcetree.c/4811
   8 2979 0          Nextnode    222.3007   26582.5664  force_treeallocate()/forcetree.c/4820
   8 2980 0            Father    221.9056   26804.4719  force_treeallocate()/forcetree.c/4827
  88 2984 0      DataNodeList     58.1818   27075.2740  subfind_density()/subfind_density.c/89
  88 2985 0              Left    126.8936   27202.1676  subfind_density()/subfind_density.c/91
 119 2990 0             DM_Vz    126.8937   27724.7193  subfind_density()/subfind_density.c/97
 119 2991 0    	  DensDataGet    114.4174   27839.1367  subfind_density()/subfind_density.c/161
 119 2992 0  	  DensDataResult     85.8130   27924.9497  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------   8 2981 0           Ngblist     63.4063   26867.8783  subfind_density()/subfind_density.c/78
   8 2982 0         Dist2list    126.8127   26994.6910  subfind_density()/subfind_density.c/79
   8 2983 0    DataIndexTable     21.8182   27016.5092  subfind_density()/subfind_density.c/87
  88 2986 0             Right    126.8936   27329.0613  subfind_density()/subfind_density.c/92
  88 2987 0              Todo     15.8617   27344.9230  subfind_density()/subfind_density.c/93


Task=119: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.0503 MB).
task 119: endrun called with an error level of 812


   8 2984 0      DataNodeList     58.1818   27074.6910  subfind_density()/subfind_density.c/89
   8 2985 0              Left    126.8127   27201.5037  subfind_density()/subfind_density.c/91
   8 2986 0             Right    126.8127   27328.3164  subfind_density()/subfind_density.c/92
  88 2988 0             DM_Vx    126.8936   27471.8166  subfind_density()/subfind_density.c/95
  88 2989 0             DM_Vy    126.8936   27598.7102  subfind_density()/subfind_density.c/96
  88 2990 0             DM_Vz    126.8936   27725.6038  subfind_density()/subfind_density.c/97
  88 2991 0    	  DensDataGet    107.5182   27833.1220  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 119
   8 2987 0              Todo     15.8516   27344.1679  subfind_density()/subfind_density.c/93
  88 2992 0  	  DensDataResult     80.6386   27913.7607  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   8 2988 0             DM_Vx    126.8127   27470.9806  subfind_density()/subfind_density.c/95

Task=88: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.2393 MB).
task 88: endrun called with an error level of 812


   8 2989 0             DM_Vy    126.8127   27597.7933  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 88
   8 2990 0             DM_Vz    126.8127   27724.6060  subfind_density()/subfind_density.c/97
task 88: endrun called with an error level of 124


   8 2991 0    	  DensDataGet    116.7704   27841.3764  subfind_density()/subfind_density.c/161
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 88:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 88
   8 2992 0  	  DensDataResult     87.5778   27928.9542  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=8: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.0458 MB).
task 8: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 8
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 8:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 8: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 8
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  90 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  90 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  90 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  90 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  90 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  90 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  90 2973 0         FOF_PList    380.6831   18861.4794  fof_fof()/fof.c/249
  90 2974 0         FOF_GList    634.4719   19495.9513  fof_fof()/fof.c/337
  90 2975 0             Group     10.9840   19506.9353  fof_fof()/fof.c/380
  37 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  37 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  37 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  37 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  37 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  37 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  37 2973 0         FOF_PList    380.6883   18861.4845  fof_fof()/fof.c/249
  37 2974 0         FOF_GList    634.4805   19495.9651  fof_fof()/fof.c/337
  37 2975 0             Group     13.1609   19509.1259  fof_fof()/fof.c/380
  37 2976 0   DomainNodeIndex      0.3458   19509.4717  force_treeallocate()/forcetree.c/4800
  37 2977 0        Nodes_base   3737.4974   23246.9691  force_treeallocate()/forcetree.c/4803
  37 2978 0     Extnodes_base   3114.5811   26361.5502  force_treeallocate()/forcetree.c/4811
  37 2979 0          Nextnode    222.3007   26583.8510  force_treeallocate()/forcetree.c/4820
  37 2980 0            Father    221.9056   26805.7565  force_treeallocate()/forcetree.c/4827
  37 2981 0           Ngblist     63.4481   26869.2046  subfind_density()/subfind_density.c/78
  37 2982 0         Dist2list    126.8961   26996.1007  subfind_density()/subfind_density.c/79
  37 2983 0    DataIndexTable     21.8182   27017.9189  subfind_density()/subfind_density.c/87
  37 2984 0      DataNodeList     58.1818   27076.1007  subfind_density()/subfind_density.c/89
  90 2976 0   DomainNodeIndex      0.3458   19507.2810  force_treeallocate()/forcetree.c/4800
  90 2977 0        Nodes_base   3737.4974   23244.7784  force_treeallocate()/forcetree.c/4803
  90 2978 0     Extnodes_base   3114.5811   26359.3596  force_treeallocate()/forcetree.c/4811
  37 2985 0              Left    126.8961   27202.9968  subfind_density()/subfind_density.c/91
  37 2986 0             Right    126.8961   27329.8929  subfind_density()/subfind_density.c/92
  37 2987 0              Todo     15.8620   27345.7549  subfind_density()/subfind_density.c/93
  37 2988 0             DM_Vx    126.8961   27472.6510  subfind_density()/subfind_density.c/95
  37 2989 0             DM_Vy    126.8961   27599.5471  subfind_density()/subfind_density.c/96
  37 2990 0             DM_Vz    126.8961   27726.4432  subfind_density()/subfind_density.c/97
  90 2979 0          Nextnode    222.3007   26581.6603  force_treeallocate()/forcetree.c/4820
  90 2980 0            Father    221.9056   26803.5659  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  37 2991 0    	  DensDataGet    118.7065   27845.1497  subfind_density()/subfind_density.c/161
  37 2992 0  	  DensDataResult     89.0299   27934.1796  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  90 2981 0           Ngblist     63.4472   26867.0131  subfind_density()/subfind_density.c/78
  90 2982 0         Dist2list    126.8944   26993.9074  subfind_density()/subfind_density.c/79
  90 2983 0    DataIndexTable     21.8182   27015.7256  subfind_density()/subfind_density.c/87
 104 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 104 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 104 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 104 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 104 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67

Task=37: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.8204 MB).
task 37: endrun called with an error level of 812


  23 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  23 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  23 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  23 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  23 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  23 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  23 2973 0         FOF_PList    380.5895   18861.3857  fof_fof()/fof.c/249
  23 2974 0         FOF_GList    634.3158   19495.7015  fof_fof()/fof.c/337
  23 2975 0             Group     12.3059   19508.0074  fof_fof()/fof.c/380
  23 2976 0   DomainNodeIndex      0.3458   19508.3532  force_treeallocate()/forcetree.c/4800
  23 2977 0        Nodes_base   3737.4974   23245.8505  force_treeallocate()/forcetree.c/4803
  90 2984 0      DataNodeList     58.1818   27073.9074  subfind_density()/subfind_density.c/89
  90 2985 0              Left    126.8944   27200.8018  subfind_density()/subfind_density.c/91
  90 2986 0             Right    126.8944   27327.6962  subfind_density()/subfind_density.c/92
 104 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 104 2973 0         FOF_PList    380.5741   18861.3703  fof_fof()/fof.c/249
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 37
  23 2978 0     Extnodes_base   3114.5811   26360.4317  force_treeallocate()/forcetree.c/4811
  23 2979 0          Nextnode    222.3007   26582.7324  force_treeallocate()/forcetree.c/4820
  23 2980 0            Father    221.9056   26804.6380  force_treeallocate()/forcetree.c/4827
  23 2981 0           Ngblist     63.4316   26868.0696  subfind_density()/subfind_density.c/78
  90 2987 0              Todo     15.8618   27343.5580  subfind_density()/subfind_density.c/93
 104 2974 0         FOF_GList    634.2901   19495.6604  fof_fof()/fof.c/337
 104 2975 0             Group     11.8328   19507.4931  fof_fof()/fof.c/380
 104 2976 0   DomainNodeIndex      0.3458   19507.8389  force_treeallocate()/forcetree.c/4800
 104 2977 0        Nodes_base   3737.4974   23245.3363  force_treeallocate()/forcetree.c/4803
  23 2982 0         Dist2list    126.8632   26994.9328  subfind_density()/subfind_density.c/79
  23 2983 0    DataIndexTable     21.8182   27016.7509  subfind_density()/subfind_density.c/87
  23 2984 0      DataNodeList     58.1818   27074.9327  subfind_density()/subfind_density.c/89
  90 2988 0             DM_Vx    126.8944   27470.4523  subfind_density()/subfind_density.c/95
 104 2978 0     Extnodes_base   3114.5811   26359.9174  force_treeallocate()/forcetree.c/4811
 104 2979 0          Nextnode    222.3007   26582.2181  force_treeallocate()/forcetree.c/4820
 104 2980 0            Father    221.9056   26804.1237  force_treeallocate()/forcetree.c/4827
 104 2981 0           Ngblist     63.4290   26867.5527  subfind_density()/subfind_density.c/78
 104 2982 0         Dist2list    126.8580   26994.4108  subfind_density()/subfind_density.c/79
  23 2985 0              Left    126.8632   27201.7959  subfind_density()/subfind_density.c/91
  23 2986 0             Right    126.8632   27328.6590  subfind_density()/subfind_density.c/92
  23 2987 0              Todo     15.8579   27344.5169  subfind_density()/subfind_density.c/93
  23 2988 0             DM_Vx    126.8632   27471.3801  subfind_density()/subfind_density.c/95
  90 2989 0             DM_Vy    126.8944   27597.3467  subfind_density()/subfind_density.c/96
 104 2983 0    DataIndexTable     21.8182   27016.2289  subfind_density()/subfind_density.c/87
 104 2984 0      DataNodeList     58.1818   27074.4107  subfind_density()/subfind_density.c/89
  23 2989 0             DM_Vy    126.8632   27598.2433  subfind_density()/subfind_density.c/96
  23 2990 0             DM_Vz    126.8632   27725.1064  subfind_density()/subfind_density.c/97
  23 2991 0    	  DensDataGet    130.1603   27855.2667  subfind_density()/subfind_density.c/161
  90 2990 0             DM_Vz    126.8944   27724.2411  subfind_density()/subfind_density.c/97
 104 2985 0              Left    126.8580   27201.2687  subfind_density()/subfind_density.c/91
 104 2986 0             Right    126.8580   27328.1268  subfind_density()/subfind_density.c/92
  23 2992 0  	  DensDataResult     97.6202   27952.8869  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=23: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=47.1131 MB).
task 23: endrun called with an error level of 812


  90 2991 0    	  DensDataGet    125.1476   27849.3887  subfind_density()/subfind_density.c/161
 104 2987 0              Todo     15.8573   27343.9840  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 23
  90 2992 0  	  DensDataResult     93.8607   27943.2494  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 104 2988 0             DM_Vx    126.8580   27470.8420  subfind_density()/subfind_density.c/95

Task=90: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.7506 MB).
 104 2989 0             DM_Vy    126.8580   27597.7001  subfind_density()/subfind_density.c/96
task 90: endrun called with an error level of 812


 104 2990 0             DM_Vz    126.8580   27724.5581  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 90
 104 2991 0    	  DensDataGet    130.9355   27855.4936  subfind_density()/subfind_density.c/161
 104 2992 0  	  DensDataResult     98.2016   27953.6952  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=104: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.3048 MB).
task 104: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 104
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 117 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 117 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 117 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 117 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 117 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 117 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
 117 2973 0         FOF_PList    380.4264   18861.2226  fof_fof()/fof.c/249
 117 2974 0         FOF_GList    634.0439   19495.2665  fof_fof()/fof.c/337
 117 2975 0             Group     10.6260   19505.8925  fof_fof()/fof.c/380
 117 2976 0   DomainNodeIndex      0.3458   19506.2383  force_treeallocate()/forcetree.c/4800
 117 2977 0        Nodes_base   3737.4974   23243.7357  force_treeallocate()/forcetree.c/4803
 117 2978 0     Extnodes_base   3114.5811   26358.3168  force_treeallocate()/forcetree.c/4811
 117 2979 0          Nextnode    222.3007   26580.6176  force_treeallocate()/forcetree.c/4820
task 104: endrun called with an error level of 124


  10 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  10 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  10 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  10 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  10 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  10 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  10 2973 0         FOF_PList    380.6868   18861.4831  fof_fof()/fof.c/249
  10 2974 0         FOF_GList    634.4781   19495.9612  fof_fof()/fof.c/337
  10 2975 0             Group     12.0372   19507.9983  fof_fof()/fof.c/380
  10 2976 0   DomainNodeIndex      0.3458   19508.3441  force_treeallocate()/forcetree.c/4800
 117 2980 0            Father    221.9056   26802.5231  force_treeallocate()/forcetree.c/4827
 117 2981 0           Ngblist     63.4044   26865.9275  subfind_density()/subfind_density.c/78
 117 2982 0         Dist2list    126.8088   26992.7363  subfind_density()/subfind_density.c/79
 117 2983 0    DataIndexTable     21.8182   27014.5545  subfind_density()/subfind_density.c/87
 117 2984 0      DataNodeList     58.1818   27072.7363  subfind_density()/subfind_density.c/89
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 104:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 104
  10 2977 0        Nodes_base   3737.4974   23245.8415  force_treeallocate()/forcetree.c/4803
  10 2978 0     Extnodes_base   3114.5811   26360.4226  force_treeallocate()/forcetree.c/4811
  10 2979 0          Nextnode    222.3007   26582.7233  force_treeallocate()/forcetree.c/4820
 117 2985 0              Left    126.8088   27199.5451  subfind_density()/subfind_density.c/91
 117 2986 0             Right    126.8088   27326.3539  subfind_density()/subfind_density.c/92
 117 2987 0              Todo     15.8511   27342.2050  subfind_density()/subfind_density.c/93
  10 2980 0            Father    221.9056   26804.6289  force_treeallocate()/forcetree.c/4827
  10 2981 0           Ngblist     63.4478   26868.0767  subfind_density()/subfind_density.c/78
 117 2988 0             DM_Vx    126.8088   27469.0138  subfind_density()/subfind_density.c/95
 117 2989 0             DM_Vy    126.8088   27595.8225  subfind_density()/subfind_density.c/96
 117 2990 0             DM_Vz    126.8088   27722.6313  subfind_density()/subfind_density.c/97
  10 2982 0         Dist2list    126.8956   26994.9723  subfind_density()/subfind_density.c/79
  10 2983 0    DataIndexTable     21.8182   27016.7905  subfind_density()/subfind_density.c/87
  10 2984 0      DataNodeList     58.1818   27074.9723  subfind_density()/subfind_density.c/89
  10 2985 0              Left    126.8956   27201.8679  subfind_density()/subfind_density.c/91
 117 2991 0    	  DensDataGet    117.4520   27840.0833  subfind_density()/subfind_density.c/161
 117 2992 0  	  DensDataResult     88.0890   27928.1723  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=117: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.8277 MB).
  10 2986 0             Right    126.8956   27328.7635  subfind_density()/subfind_density.c/92
  10 2987 0              Todo     15.8620   27344.6255  subfind_density()/subfind_density.c/93
  10 2988 0             DM_Vx    126.8956   27471.5211  subfind_density()/subfind_density.c/95
task 117: endrun called with an error level of 812


  10 2989 0             DM_Vy    126.8956   27598.4167  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 117
  10 2990 0             DM_Vz    126.8956   27725.3123  subfind_density()/subfind_density.c/97
  10 2991 0    	  DensDataGet    119.0067   27844.3190  subfind_density()/subfind_density.c/161
  10 2992 0  	  DensDataResult     89.2550   27933.5740  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=10: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.426 MB).
task 10: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 10
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  36 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  36 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  36 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  36 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  36 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  36 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  36 2973 0         FOF_PList    380.5846   18861.3809  fof_fof()/fof.c/249
  36 2974 0         FOF_GList    634.3077   19495.6886  fof_fof()/fof.c/337
  36 2975 0             Group     13.1505   19508.8391  fof_fof()/fof.c/380
  36 2976 0   DomainNodeIndex      0.3458   19509.1848  force_treeallocate()/forcetree.c/4800
  36 2977 0        Nodes_base   3737.4974   23246.6822  force_treeallocate()/forcetree.c/4803
  36 2978 0     Extnodes_base   3114.5811   26361.2634  force_treeallocate()/forcetree.c/4811
  36 2979 0          Nextnode    222.3007   26583.5641  force_treeallocate()/forcetree.c/4820
  36 2980 0            Father    221.9056   26805.4697  force_treeallocate()/forcetree.c/4827
  36 2981 0           Ngblist     63.4308   26868.9005  subfind_density()/subfind_density.c/78
  36 2982 0         Dist2list    126.8615   26995.7620  subfind_density()/subfind_density.c/79
  36 2983 0    DataIndexTable     21.8182   27017.5802  subfind_density()/subfind_density.c/87
  36 2984 0      DataNodeList     58.1818   27075.7620  subfind_density()/subfind_density.c/89
  36 2985 0              Left    126.8615   27202.6235  subfind_density()/subfind_density.c/91
  36 2986 0             Right    126.8615   27329.4850  subfind_density()/subfind_density.c/92
  36 2987 0              Todo     15.8577   27345.3427  subfind_density()/subfind_density.c/93
  36 2988 0             DM_Vx    126.8615   27472.2043  subfind_density()/subfind_density.c/95
  36 2989 0             DM_Vy    126.8615   27599.0658  subfind_density()/subfind_density.c/96
  36 2990 0             DM_Vz    126.8615   27725.9274  subfind_density()/subfind_density.c/97
  36 2991 0    	  DensDataGet    130.4498   27856.3772  subfind_density()/subfind_density.c/161
  36 2992 0  	  DensDataResult     97.8374   27954.2146  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=36: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.7854 MB).
task 36: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 36
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  11 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  11 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  11 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  11 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  11 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  11 2972 0          TopNodes      5.1369   18480.7962  domain_allocate()/domain.c/570
  11 2973 0         FOF_PList    380.6637   18861.4599  fof_fof()/fof.c/249
  11 2974 0         FOF_GList    634.4395   19495.8994  fof_fof()/fof.c/337
  11 2975 0             Group     11.3928   19507.2923  fof_fof()/fof.c/380
  11 2976 0   DomainNodeIndex      0.3458   19507.6380  force_treeallocate()/forcetree.c/4800
  11 2977 0        Nodes_base   3737.4974   23245.1354  force_treeallocate()/forcetree.c/4803
  11 2978 0     Extnodes_base   3114.5811   26359.7165  force_treeallocate()/forcetree.c/4811
  11 2979 0          Nextnode    222.3007   26582.0173  force_treeallocate()/forcetree.c/4820
  11 2980 0            Father    221.9056   26803.9229  force_treeallocate()/forcetree.c/4827
  11 2981 0           Ngblist     63.4440   26867.3668  subfind_density()/subfind_density.c/78
  11 2982 0         Dist2list    126.8879   26994.2547  subfind_density()/subfind_density.c/79
  11 2983 0    DataIndexTable     21.8182   27016.0729  subfind_density()/subfind_density.c/87
  11 2984 0      DataNodeList     58.1818   27074.2547  subfind_density()/subfind_density.c/89
  11 2985 0              Left    126.8879   27201.1426  subfind_density()/subfind_density.c/91
  11 2986 0             Right    126.8879   27328.0305  subfind_density()/subfind_density.c/92
  11 2987 0              Todo     15.8610   27343.8915  subfind_density()/subfind_density.c/93
  11 2988 0             DM_Vx    126.8879   27470.7794  subfind_density()/subfind_density.c/95
  11 2989 0             DM_Vy    126.8879   27597.6673  subfind_density()/subfind_density.c/96
  11 2990 0             DM_Vz    126.8879   27724.5552  subfind_density()/subfind_density.c/97
  11 2991 0    	  DensDataGet    115.5153   27840.0705  subfind_density()/subfind_density.c/161
  11 2992 0  	  DensDataResult     86.6365   27926.7070  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=11: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.293 MB).
task 11: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 11
-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63110.11 Mb (on task=120), Smallest =   62842.05 Mb (on task=0), Average =   63040.06 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1689.72 Mb (on task=0), Smallest =    1421.95 Mb (on task=120), Average =    1492.02 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.11 Mb
AllocMem:	 Largest =    1689.72 Mb (on task=0), Smallest =    1421.95 Mb (on task=120), Average =    1492.02 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_022_z000p247/snap_022_z000p247.0.hdf5' on task=0 (contains 134665512 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  134530635  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      9224  (tot=     0000134208) masstab=0
Type 3 (bulge):   125653  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.814267

Setting next time for line of sight file to Time_next= 0.803424

Setting next time for snipshot file to Time_next= 0.810044

Setting next time for fof output to Time_next= 0.810044
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=12.1854 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=102753
After=103465
NTopleaves= 90532  NTopnodes=103465 (space for 465370)
gravity work-load balance=1.00037   memory-balance=1.00037   SPH work-load balance=0
iter=0 exchange of 2108046633 particles (ret=0)
domain decomposition done. (took 38.7715 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5667 MByte in top-level domain structure

Allocated 7296.26 MByte for BH-tree, and 0.345352 Mbyte for top-leaves.  (presently allocted 25777.4 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.3 Mbyte (on task=94), Smallest = 26119.3 Mbyte, Average = 26120.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  94 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  94 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  94 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  94 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  94 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  94 2972 0          TopNodes      5.1309   18480.7903  domain_allocate()/domain.c/570
  94 2973 0   DomainNodeIndex      0.3454   18481.1356  force_treeallocate()/forcetree.c/4800
  94 2974 0        Nodes_base   3737.4864   22218.6220  force_treeallocate()/forcetree.c/4803
  94 2975 0     Extnodes_base   3114.5720   25333.1940  force_treeallocate()/forcetree.c/4811
  94 2976 0          Nextnode    222.3003   25555.4943  force_treeallocate()/forcetree.c/4820
  94 2977 0            Father    221.9056   25777.3999  force_treeallocate()/forcetree.c/4827
  94 2978 0           Ngblist     63.4250   25840.8249  density()/density.c/690
  94 2979 0              Left    126.8499   25967.6748  density()/density.c/692
  94 2980 0             Right    126.8499   26094.5247  density()/density.c/693
  94 2981 0    DataIndexTable      7.0175   26101.5423  density()/density.c/738
  94 2982 0      DataNodeList     18.7134   26120.2557  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.211.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.271.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.4 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=94353
After=103769
NTopleaves= 90798  NTopnodes=103769 (space for 465370)
gravity work-load balance=1.00037   memory-balance=1.00037   SPH work-load balance=0
iter=0 exchange of 2117058561 particles (ret=0)
domain decomposition done. (took 31.8514 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5528 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.8 MB)
Tree construction.

Start linking particles (presently allocated=26538.5 MB)
links on local processor done (took 173.988 sec).
Marked=0217827951 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26862.5 MB) 
have done 0002838269 cross links (processed 0217827951, took 27.7897 sec)
have done 0001249728 cross links (processed 0179588867, took 26.3031 sec)
have done 0000893088 cross links (processed 0156199957, took 20.7971 sec)
have done 0000571995 cross links (processed 0106951947, took 23.196 sec)
have done 0000363563 cross links (processed 0068847860, took 21.8511 sec)
have done 0000243912 cross links (processed 0050264032, took 18.1418 sec)
have done 0000159364 cross links (processed 0031419139, took 18.1989 sec)
have done 0000095751 cross links (processed 0022525064, took 9.46361 sec)
have done 0000058021 cross links (processed 0013206482, took 18.1233 sec)
have done 0000033960 cross links (processed 0001845728, took 0.67604 sec)
have done 0000017642 cross links (processed 0000712400, took 0.45187 sec)
have done 0000008172 cross links (processed 0000268664, took 0.35745 sec)
have done 0000003134 cross links (processed 0000068583, took 0.205899 sec)
have done 0000001296 cross links (processed 0000024465, took 0.182599 sec)
have done 0000000706 cross links (processed 0000022229, took 0.311205 sec)
have done 0000000240 cross links (processed 0000004203, took 0.361473 sec)
have done 0000000087 cross links (processed 0000001463, took 0.36126 sec)
have done 0000000043 cross links (processed 0000000447, took 0.360149 sec)
have done 0000000021 cross links (processed 0000000147, took 0.360016 sec)
have done 0000000018 cross links (processed 0000000075, took 0.360954 sec)
have done 0000000014 cross links (processed 0000000076, took 0.359971 sec)
have done 0000000011 cross links (processed 0000000080, took 0.360244 sec)
have done 0000000002 cross links (processed 0000000029, took 0.360401 sec)
have done 0000000000 cross links (processed 0000000004, took 0.360244 sec)
Local groups found.

group finding took = 363.859 sec
Start finding nearest dm-particle (presently allocated=26538.5 MB)
still finding nearest... (presently allocated=26805.9 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.810266 sec
compiling local group data and catalogue took = 11.9778 sec

Total number of groups with at least 32 particles: 1324058
Largest group has 130329277 particles.
Total number of particles in groups: 1262863805

group properties are now allocated.. (presently allocated=19507.1 MB)
computation of group properties took = 0.532205 sec
start global sorting of group catalogues number 22
LargestGroup = 101596.753931; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.222107 sec
Group catalogues globally sorted. took = 7.39667 sec
starting saving of group catalogue number 22
data//groups_022_z000p247
Group catalogues number 22 saved. took = 16.1864 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 7.14421 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 2.86046 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 12.284 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  91 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  91 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  91 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  91 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  91 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  91 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  91 2973 0         FOF_PList    380.5305   18861.3359  fof_fof()/fof.c/249
  91 2974 0         FOF_GList    634.2176   19495.5535  fof_fof()/fof.c/337
  91 2975 0             Group     13.2678   19508.8212  fof_fof()/fof.c/380
  91 2976 0   DomainNodeIndex      0.3464   19509.1676  force_treeallocate()/forcetree.c/4800
  91 2977 0        Nodes_base   3737.5142   23246.6818  force_treeallocate()/forcetree.c/4803
  91 2978 0     Extnodes_base   3114.5952   26361.2770  force_treeallocate()/forcetree.c/4811
  91 2979 0          Nextnode    222.3014   26583.5784  force_treeallocate()/forcetree.c/4820
  91 2980 0            Father    221.9056   26805.4840  force_treeallocate()/forcetree.c/4827
  91 2981 0           Ngblist     63.4218   26868.9058  subfind_density()/subfind_density.c/78
  91 2982 0         Dist2list    126.8435   26995.7493  subfind_density()/subfind_density.c/79
  91 2983 0    DataIndexTable     21.8182   27017.5675  subfind_density()/subfind_density.c/87
  91 2984 0      DataNodeList     58.1818   27075.7493  subfind_density()/subfind_density.c/89
  36 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  36 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  36 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  36 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  36 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  36 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  36 2973 0         FOF_PList    380.5126   18861.3180  fof_fof()/fof.c/249
  36 2974 0         FOF_GList    634.1877   19495.5056  fof_fof()/fof.c/337
  36 2975 0             Group     10.7339   19506.2396  fof_fof()/fof.c/380
  36 2976 0   DomainNodeIndex      0.3464   19506.5859  force_treeallocate()/forcetree.c/4800
  91 2985 0              Left    126.8435   27202.5928  subfind_density()/subfind_density.c/91
  91 2986 0             Right    126.8435   27329.4363  subfind_density()/subfind_density.c/92
  91 2987 0              Todo     15.8554   27345.2917  subfind_density()/subfind_density.c/93
  91 2988 0             DM_Vx    126.8435   27472.1353  subfind_density()/subfind_density.c/95
  91 2989 0             DM_Vy    126.8435   27598.9788  subfind_density()/subfind_density.c/96
  91 2990 0             DM_Vz    126.8435   27725.8223  subfind_density()/subfind_density.c/97
  91 2991 0    	  DensDataGet    114.4114   27840.2337  subfind_density()/subfind_density.c/161
  36 2977 0        Nodes_base   3737.5142   23244.1002  force_treeallocate()/forcetree.c/4803
  36 2978 0     Extnodes_base   3114.5952   26358.6953  force_treeallocate()/forcetree.c/4811
  36 2979 0          Nextnode    222.3014   26580.9968  force_treeallocate()/forcetree.c/4820
  91 2992 0  	  DensDataResult     85.8086   27926.0423  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=91: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.9577 MB).
task 91: endrun called with an error level of 812


  36 2980 0            Father    221.9056   26802.9024  force_treeallocate()/forcetree.c/4827
  36 2981 0           Ngblist     63.4188   26866.3211  subfind_density()/subfind_density.c/78
  36 2982 0         Dist2list    126.8375   26993.1587  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 91
  36 2983 0    DataIndexTable     21.8182   27014.9768  subfind_density()/subfind_density.c/87
  36 2984 0      DataNodeList     58.1818   27073.1586  subfind_density()/subfind_density.c/89
  36 2985 0              Left    126.8375   27199.9962  subfind_density()/subfind_density.c/91
  36 2986 0             Right    126.8375   27326.8337  subfind_density()/subfind_density.c/92
  36 2987 0              Todo     15.8547   27342.6884  subfind_density()/subfind_density.c/93
  36 2988 0             DM_Vx    126.8375   27469.5259  subfind_density()/subfind_density.c/95
  36 2989 0             DM_Vy    126.8375   27596.3635  subfind_density()/subfind_density.c/96
  36 2990 0             DM_Vz    126.8375   27723.2010  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  36 2991 0    	  DensDataGet    113.4726   27836.6736  subfind_density()/subfind_density.c/161
  88 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  88 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  88 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  88 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  88 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  88 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  88 2973 0         FOF_PList    380.5446   18861.3500  fof_fof()/fof.c/249
  88 2974 0         FOF_GList    634.2410   19495.5909  fof_fof()/fof.c/337
  88 2975 0             Group     12.3547   19507.9456  fof_fof()/fof.c/380
  88 2976 0   DomainNodeIndex      0.3464   19508.2920  force_treeallocate()/forcetree.c/4800
  88 2977 0        Nodes_base   3737.5142   23245.8062  force_treeallocate()/forcetree.c/4803
  36 2992 0  	  DensDataResult     85.1044   27921.7780  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  88 2978 0     Extnodes_base   3114.5952   26360.4014  force_treeallocate()/forcetree.c/4811
  88 2979 0          Nextnode    222.3014   26582.7028  force_treeallocate()/forcetree.c/4820
  88 2980 0            Father    221.9056   26804.6084  force_treeallocate()/forcetree.c/4827
  88 2981 0           Ngblist     63.4241   26868.0325  subfind_density()/subfind_density.c/78

Task=36: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.222 MB).
task 36: endrun called with an error level of 812


  39 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  39 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  39 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  39 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  39 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  39 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  39 2973 0         FOF_PList    380.2931   18861.0984  fof_fof()/fof.c/249
  39 2974 0         FOF_GList    633.8218   19494.9202  fof_fof()/fof.c/337
  39 2975 0             Group     11.6761   19506.5963  fof_fof()/fof.c/380
  39 2976 0   DomainNodeIndex      0.3464   19506.9427  force_treeallocate()/forcetree.c/4800
  39 2977 0        Nodes_base   3737.5142   23244.4569  force_treeallocate()/forcetree.c/4803
  39 2978 0     Extnodes_base   3114.5952   26359.0521  force_treeallocate()/forcetree.c/4811
  39 2979 0          Nextnode    222.3014   26581.3535  force_treeallocate()/forcetree.c/4820
  39 2980 0            Father    221.9056   26803.2591  force_treeallocate()/forcetree.c/4827
  88 2982 0         Dist2list    126.8482   26994.8807  subfind_density()/subfind_density.c/79
  88 2983 0    DataIndexTable     21.8182   27016.6989  subfind_density()/subfind_density.c/87
  88 2984 0      DataNodeList     58.1818   27074.8807  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 36
  39 2981 0           Ngblist     63.3822   26866.6413  subfind_density()/subfind_density.c/78
  39 2982 0         Dist2list    126.7644   26993.4056  subfind_density()/subfind_density.c/79
  39 2983 0    DataIndexTable     21.8182   27015.2238  subfind_density()/subfind_density.c/87
  39 2984 0      DataNodeList     58.1818   27073.4056  subfind_density()/subfind_density.c/89
  88 2985 0              Left    126.8482   27201.7289  subfind_density()/subfind_density.c/91
  88 2986 0             Right    126.8482   27328.5771  subfind_density()/subfind_density.c/92
  39 2985 0              Left    126.7644   27200.1700  subfind_density()/subfind_density.c/91
  39 2986 0             Right    126.7644   27326.9343  subfind_density()/subfind_density.c/92
  39 2987 0              Todo     15.8456   27342.7799  subfind_density()/subfind_density.c/93
  88 2987 0              Todo     15.8560   27344.4331  subfind_density()/subfind_density.c/93
  88 2988 0             DM_Vx    126.8482   27471.2813  subfind_density()/subfind_density.c/95
  88 2989 0             DM_Vy    126.8482   27598.1295  subfind_density()/subfind_density.c/96
  39 2988 0             DM_Vx    126.7644   27469.5442  subfind_density()/subfind_density.c/95
  39 2989 0             DM_Vy    126.7644   27596.3086  subfind_density()/subfind_density.c/96
  39 2990 0             DM_Vz    126.7644   27723.0730  subfind_density()/subfind_density.c/97
  88 2990 0             DM_Vz    126.8482   27724.9777  subfind_density()/subfind_density.c/97
  39 2991 0    	  DensDataGet    117.9990   27841.0719  subfind_density()/subfind_density.c/161
  39 2992 0  	  DensDataResult     88.4992   27929.5711  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=39: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.4289 MB).
  88 2991 0    	  DensDataGet    127.6443   27852.6220  subfind_density()/subfind_density.c/161
task 39: endrun called with an error level of 812


  88 2992 0  	  DensDataResult     95.7332   27948.3552  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 39

Task=88: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=51.6448 MB).
task 88: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 88
task 88: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 88:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 88
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 120 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 120 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 120 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 120 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 120 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 120 2973 0         FOF_PList    379.8314   18860.6368  fof_fof()/fof.c/249
 120 2974 0         FOF_GList    633.0523   19493.6891  fof_fof()/fof.c/337
 120 2975 0             Group     12.1638   19505.8528  fof_fof()/fof.c/380
 120 2976 0   DomainNodeIndex      0.3464   19506.1992  force_treeallocate()/forcetree.c/4800
 120 2977 0        Nodes_base   3737.5142   23243.7134  force_treeallocate()/forcetree.c/4803
 120 2978 0     Extnodes_base   3114.5952   26358.3086  force_treeallocate()/forcetree.c/4811
 120 2979 0          Nextnode    222.3014   26580.6101  force_treeallocate()/forcetree.c/4820
 120 2980 0            Father    221.9056   26802.5156  force_treeallocate()/forcetree.c/4827
 120 2981 0           Ngblist     63.3052   26865.8209  subfind_density()/subfind_density.c/78
 120 2982 0         Dist2list    126.6105   26992.4313  subfind_density()/subfind_density.c/79
 120 2983 0    DataIndexTable     21.8182   27014.2495  subfind_density()/subfind_density.c/87
 120 2984 0      DataNodeList     58.1818   27072.4313  subfind_density()/subfind_density.c/89
 120 2985 0              Left    126.6105   27199.0418  subfind_density()/subfind_density.c/91
 120 2986 0             Right    126.6105   27325.6522  subfind_density()/subfind_density.c/92
 120 2987 0              Todo     15.8263   27341.4786  subfind_density()/subfind_density.c/93
 120 2988 0             DM_Vx    126.6105   27468.0890  subfind_density()/subfind_density.c/95
 120 2989 0             DM_Vy    126.6105   27594.6995  subfind_density()/subfind_density.c/96
 120 2990 0             DM_Vz    126.6105   27721.3100  subfind_density()/subfind_density.c/97
 120 2991 0    	  DensDataGet    139.9166   27861.2266  subfind_density()/subfind_density.c/161
 120 2992 0  	  DensDataResult    104.9375   27966.1640  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=120: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=33.836 MB).
task 120: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 120
  89 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  89 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  89 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  89 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  89 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  89 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  89 2973 0         FOF_PList    380.4803   18861.2857  fof_fof()/fof.c/249
  89 2974 0         FOF_GList    634.1339   19495.4196  fof_fof()/fof.c/337
  89 2975 0             Group     11.7290   19507.1486  fof_fof()/fof.c/380
  89 2976 0   DomainNodeIndex      0.3464   19507.4949  force_treeallocate()/forcetree.c/4800
  89 2977 0        Nodes_base   3737.5142   23245.0091  force_treeallocate()/forcetree.c/4803
  89 2978 0     Extnodes_base   3114.5952   26359.6043  force_treeallocate()/forcetree.c/4811
  89 2979 0          Nextnode    222.3014   26581.9058  force_treeallocate()/forcetree.c/4820
task 120: endrun called with an error level of 124


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  89 2980 0            Father    221.9056   26803.8114  force_treeallocate()/forcetree.c/4827
  89 2981 0           Ngblist     63.4134   26867.2247  subfind_density()/subfind_density.c/78
  89 2982 0         Dist2list    126.8268   26994.0515  subfind_density()/subfind_density.c/79
  89 2983 0    DataIndexTable     21.8182   27015.8697  subfind_density()/subfind_density.c/87
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 120:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 120
  38 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  38 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  38 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  38 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  38 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  89 2984 0      DataNodeList     58.1818   27074.0515  subfind_density()/subfind_density.c/89
  89 2985 0              Left    126.8268   27200.8783  subfind_density()/subfind_density.c/91
  89 2986 0             Right    126.8268   27327.7050  subfind_density()/subfind_density.c/92
  38 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  38 2973 0         FOF_PList    380.2454   18861.0508  fof_fof()/fof.c/249
  38 2974 0         FOF_GList    633.7424   19494.7932  fof_fof()/fof.c/337
  89 2987 0              Todo     15.8533   27343.5584  subfind_density()/subfind_density.c/93
  89 2988 0             DM_Vx    126.8268   27470.3852  subfind_density()/subfind_density.c/95
  89 2989 0             DM_Vy    126.8268   27597.2119  subfind_density()/subfind_density.c/96
  38 2975 0             Group     13.3777   19508.1710  fof_fof()/fof.c/380
  38 2976 0   DomainNodeIndex      0.3464   19508.5173  force_treeallocate()/forcetree.c/4800
  38 2977 0        Nodes_base   3737.5142   23246.0315  force_treeallocate()/forcetree.c/4803
  89 2990 0             DM_Vz    126.8268   27724.0387  subfind_density()/subfind_density.c/97
  89 2991 0    	  DensDataGet    144.1175   27868.1562  subfind_density()/subfind_density.c/161
  89 2992 0  	  DensDataResult    108.0881   27976.2443  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  38 2978 0     Extnodes_base   3114.5952   26360.6267  force_treeallocate()/forcetree.c/4811
  38 2979 0          Nextnode    222.3014   26582.9282  force_treeallocate()/forcetree.c/4820
  38 2980 0            Father    221.9056   26804.8338  force_treeallocate()/forcetree.c/4827

Task=89: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=23.7557 MB).
task 89: endrun called with an error level of 812


  38 2981 0           Ngblist     63.3742   26868.2080  subfind_density()/subfind_density.c/78
  38 2982 0         Dist2list    126.7485   26994.9565  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 89
  38 2983 0    DataIndexTable     21.8182   27016.7747  subfind_density()/subfind_density.c/87
  38 2984 0      DataNodeList     58.1818   27074.9565  subfind_density()/subfind_density.c/89
  38 2985 0              Left    126.7485   27201.7049  subfind_density()/subfind_density.c/91
  38 2986 0             Right    126.7485   27328.4534  subfind_density()/subfind_density.c/92
  38 2987 0              Todo     15.8436   27344.2970  subfind_density()/subfind_density.c/93
  38 2988 0             DM_Vx    126.7485   27471.0455  subfind_density()/subfind_density.c/95
  38 2989 0             DM_Vy    126.7485   27597.7939  subfind_density()/subfind_density.c/96
  38 2990 0             DM_Vz    126.7485   27724.5424  subfind_density()/subfind_density.c/97
  38 2991 0    	  DensDataGet    121.8907   27846.4331  subfind_density()/subfind_density.c/161
  38 2992 0  	  DensDataResult     91.4180   27937.8511  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=38: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.1489 MB).
task 38: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 38
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   0 2965 0     ProcessedFlag     55.4764      57.4321  allocate_memory()/allocate.c/52
   0 2966 0  NextActiveParticle    221.9056     279.3377  allocate_memory()/allocate.c/55
   0 2967 0     NextInTimeBin    221.9056     501.2433  allocate_memory()/allocate.c/58
   0 2968 0     PrevInTimeBin    221.9056     723.1489  allocate_memory()/allocate.c/61
   0 2969 0                 P  17752.4469   18475.5958  allocate_memory()/allocate.c/67
   0 2972 0          TopNodes      5.1460   18480.8043  domain_allocate()/domain.c/570
   0 2973 0         FOF_PList    380.4956   18861.2999  fof_fof()/fof.c/249
   0 2974 0         FOF_GList    634.1594   19495.4593  fof_fof()/fof.c/337
   0 2975 0             Group     11.5941   19507.0535  fof_fof()/fof.c/380
   0 2976 0   DomainNodeIndex      0.3464   19507.3998  force_treeallocate()/forcetree.c/4800
   0 2977 0        Nodes_base   3737.5142   23244.9140  force_treeallocate()/forcetree.c/4803
   0 2978 0     Extnodes_base   3114.5952   26359.5092  force_treeallocate()/forcetree.c/4811
   0 2979 0          Nextnode    222.3014   26581.8107  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 127 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 127 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 127 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   0 2980 0            Father    221.9056   26803.7163  force_treeallocate()/forcetree.c/4827
   0 2981 0           Ngblist     63.4159   26867.1322  subfind_density()/subfind_density.c/78
   0 2982 0         Dist2list    126.8319   26993.9641  subfind_density()/subfind_density.c/79
   0 2983 0    DataIndexTable     21.8182   27015.7822  subfind_density()/subfind_density.c/87
   0 2984 0      DataNodeList     58.1818   27073.9640  subfind_density()/subfind_density.c/89
 127 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 127 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 127 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 127 2973 0         FOF_PList    380.5212   18861.3266  fof_fof()/fof.c/249
 127 2974 0         FOF_GList    634.2020   19495.5286  fof_fof()/fof.c/337
 127 2975 0             Group     10.7339   19506.2626  fof_fof()/fof.c/380
 127 2976 0   DomainNodeIndex      0.3464   19506.6089  force_treeallocate()/forcetree.c/4800
 127 2977 0        Nodes_base   3737.5142   23244.1232  force_treeallocate()/forcetree.c/4803
 127 2978 0     Extnodes_base   3114.5952   26358.7183  force_treeallocate()/forcetree.c/4811
 127 2979 0          Nextnode    222.3014   26581.0198  force_treeallocate()/forcetree.c/4820
 127 2980 0            Father    221.9056   26802.9254  force_treeallocate()/forcetree.c/4827
 127 2981 0           Ngblist     63.4202   26866.3456  subfind_density()/subfind_density.c/78
 127 2982 0         Dist2list    126.8404   26993.1860  subfind_density()/subfind_density.c/79
 127 2983 0    DataIndexTable     21.8182   27015.0042  subfind_density()/subfind_density.c/87
 127 2984 0      DataNodeList     58.1818   27073.1860  subfind_density()/subfind_density.c/89
 127 2985 0              Left    126.8404   27200.0264  subfind_density()/subfind_density.c/91
 127 2986 0             Right    126.8404   27326.8668  subfind_density()/subfind_density.c/92
 127 2987 0              Todo     15.8551   27342.7218  subfind_density()/subfind_density.c/93
   0 2985 0              Left    126.8319   27200.7959  subfind_density()/subfind_density.c/91
   0 2986 0             Right    126.8319   27327.6278  subfind_density()/subfind_density.c/92
 127 2988 0             DM_Vx    126.8404   27469.5622  subfind_density()/subfind_density.c/95
 127 2989 0             DM_Vy    126.8404   27596.4026  subfind_density()/subfind_density.c/96
 127 2990 0             DM_Vz    126.8404   27723.2430  subfind_density()/subfind_density.c/97
 127 2991 0    	  DensDataGet    123.0278   27846.2708  subfind_density()/subfind_density.c/161
 127 2992 0  	  DensDataResult     92.2708   27938.5416  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   0 2987 0              Todo     15.8540   27343.4818  subfind_density()/subfind_density.c/93
   0 2988 0             DM_Vx    126.8319   27470.3137  subfind_density()/subfind_density.c/95

Task=127: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.4584 MB).
task 127: endrun called with an error level of 812


   0 2989 0             DM_Vy    126.8319   27597.1455  subfind_density()/subfind_density.c/96
   0 2990 0             DM_Vz    126.8319   27723.9774  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 127
   0 2991 0    	  DensDataGet    138.1469   27862.1243  subfind_density()/subfind_density.c/161
   0 2992 0  	  DensDataResult    103.6102   27965.7345  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=0: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=34.2655 MB).
task 0: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 0
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0:
task 0: endrun called with an error level of 124


  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 0
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  90 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  90 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  90 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  37 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  37 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  37 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  37 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  37 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  37 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  37 2973 0         FOF_PList    380.4108   18861.2161  fof_fof()/fof.c/249
  37 2974 0         FOF_GList    634.0180   19495.2341  fof_fof()/fof.c/337
  90 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  90 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  90 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  90 2973 0         FOF_PList    380.5393   18861.3446  fof_fof()/fof.c/249
  90 2974 0         FOF_GList    634.2321   19495.5767  fof_fof()/fof.c/337
  90 2975 0             Group     11.1210   19506.6977  fof_fof()/fof.c/380
  90 2976 0   DomainNodeIndex      0.3464   19507.0441  force_treeallocate()/forcetree.c/4800
  37 2975 0             Group     12.6826   19507.9167  fof_fof()/fof.c/380
  37 2976 0   DomainNodeIndex      0.3464   19508.2631  force_treeallocate()/forcetree.c/4800
  37 2977 0        Nodes_base   3737.5142   23245.7773  force_treeallocate()/forcetree.c/4803
  37 2978 0     Extnodes_base   3114.5952   26360.3725  force_treeallocate()/forcetree.c/4811
  90 2977 0        Nodes_base   3737.5142   23244.5583  force_treeallocate()/forcetree.c/4803
  90 2978 0     Extnodes_base   3114.5952   26359.1535  force_treeallocate()/forcetree.c/4811
  90 2979 0          Nextnode    222.3014   26581.4549  force_treeallocate()/forcetree.c/4820
  37 2979 0          Nextnode    222.3014   26582.6739  force_treeallocate()/forcetree.c/4820
  37 2980 0            Father    221.9056   26804.5795  force_treeallocate()/forcetree.c/4827
  37 2981 0           Ngblist     63.4018   26867.9813  subfind_density()/subfind_density.c/78
  90 2980 0            Father    221.9056   26803.3605  force_treeallocate()/forcetree.c/4827
  90 2981 0           Ngblist     63.4232   26866.7837  subfind_density()/subfind_density.c/78
  90 2982 0         Dist2list    126.8464   26993.6301  subfind_density()/subfind_density.c/79
  37 2982 0         Dist2list    126.8036   26994.7849  subfind_density()/subfind_density.c/79
  37 2983 0    DataIndexTable     21.8182   27016.6031  subfind_density()/subfind_density.c/87
  37 2984 0      DataNodeList     58.1818   27074.7849  subfind_density()/subfind_density.c/89
  37 2985 0              Left    126.8036   27201.5884  subfind_density()/subfind_density.c/91
  90 2983 0    DataIndexTable     21.8182   27015.4483  subfind_density()/subfind_density.c/87
  90 2984 0      DataNodeList     58.1818   27073.6301  subfind_density()/subfind_density.c/89
  37 2986 0             Right    126.8036   27328.3920  subfind_density()/subfind_density.c/92
  37 2987 0              Todo     15.8505   27344.2425  subfind_density()/subfind_density.c/93
  37 2988 0             DM_Vx    126.8036   27471.0461  subfind_density()/subfind_density.c/95
  37 2989 0             DM_Vy    126.8036   27597.8497  subfind_density()/subfind_density.c/96
  37 2990 0             DM_Vz    126.8036   27724.6533  subfind_density()/subfind_density.c/97
  37 2991 0    	  DensDataGet    130.8559   27855.5092  subfind_density()/subfind_density.c/161
  37 2992 0  	  DensDataResult     98.1419   27953.6511  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=37: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.3489 MB).
task 37: endrun called with an error level of 812


  90 2985 0              Left    126.8464   27200.4765  subfind_density()/subfind_density.c/91
  90 2986 0             Right    126.8464   27327.3229  subfind_density()/subfind_density.c/92
  90 2987 0              Todo     15.8558   27343.1787  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 37
  90 2988 0             DM_Vx    126.8464   27470.0252  subfind_density()/subfind_density.c/95
  90 2989 0             DM_Vy    126.8464   27596.8716  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  90 2990 0             DM_Vz    126.8464   27723.7180  subfind_density()/subfind_density.c/97
  90 2991 0    	  DensDataGet    135.7364   27859.4544  subfind_density()/subfind_density.c/161
  90 2992 0  	  DensDataResult    101.8023   27961.2567  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 121 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 121 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 121 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 121 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 121 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 121 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 121 2973 0         FOF_PList    380.4292   18861.2345  fof_fof()/fof.c/249
 121 2974 0         FOF_GList    634.0486   19495.2831  fof_fof()/fof.c/337
 121 2975 0             Group     13.4120   19508.6951  fof_fof()/fof.c/380
 121 2976 0   DomainNodeIndex      0.3464   19509.0415  force_treeallocate()/forcetree.c/4800
 121 2977 0        Nodes_base   3737.5142   23246.5557  force_treeallocate()/forcetree.c/4803
 121 2978 0     Extnodes_base   3114.5952   26361.1509  force_treeallocate()/forcetree.c/4811
   6 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   6 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   6 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   6 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   6 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   6 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
   6 2973 0         FOF_PList    380.4955   18861.3008  fof_fof()/fof.c/249
   6 2974 0         FOF_GList    634.1591   19495.4599  fof_fof()/fof.c/337
   6 2975 0             Group     12.0538   19507.5137  fof_fof()/fof.c/380

Task=90: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=38.7433 MB).
task 90: endrun called with an error level of 812


 121 2979 0          Nextnode    222.3014   26583.4523  force_treeallocate()/forcetree.c/4820
 121 2980 0            Father    221.9056   26805.3579  force_treeallocate()/forcetree.c/4827
 121 2981 0           Ngblist     63.4049   26868.7628  subfind_density()/subfind_density.c/78
 121 2982 0         Dist2list    126.8097   26995.5725  subfind_density()/subfind_density.c/79
   6 2976 0   DomainNodeIndex      0.3464   19507.8600  force_treeallocate()/forcetree.c/4800
   6 2977 0        Nodes_base   3737.5142   23245.3743  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 90
 121 2983 0    DataIndexTable     21.8182   27017.3907  subfind_density()/subfind_density.c/87
 121 2984 0      DataNodeList     58.1818   27075.5725  subfind_density()/subfind_density.c/89
   6 2978 0     Extnodes_base   3114.5952   26359.9694  force_treeallocate()/forcetree.c/4811
   6 2979 0          Nextnode    222.3014   26582.2709  force_treeallocate()/forcetree.c/4820
   6 2980 0            Father    221.9056   26804.1765  force_treeallocate()/forcetree.c/4827
   6 2981 0           Ngblist     63.4159   26867.5924  subfind_density()/subfind_density.c/78
   6 2982 0         Dist2list    126.8318   26994.4242  subfind_density()/subfind_density.c/79
   6 2983 0    DataIndexTable     21.8182   27016.2424  subfind_density()/subfind_density.c/87
 121 2985 0              Left    126.8097   27202.3822  subfind_density()/subfind_density.c/91
 121 2986 0             Right    126.8097   27329.1919  subfind_density()/subfind_density.c/92
 121 2987 0              Todo     15.8512   27345.0431  subfind_density()/subfind_density.c/93
   6 2984 0      DataNodeList     58.1818   27074.4242  subfind_density()/subfind_density.c/89
   6 2985 0              Left    126.8318   27201.2560  subfind_density()/subfind_density.c/91
   6 2986 0             Right    126.8318   27328.0878  subfind_density()/subfind_density.c/92
 121 2988 0             DM_Vx    126.8097   27471.8529  subfind_density()/subfind_density.c/95
 121 2989 0             DM_Vy    126.8097   27598.6626  subfind_density()/subfind_density.c/96
 121 2990 0             DM_Vz    126.8097   27725.4723  subfind_density()/subfind_density.c/97
   6 2987 0              Todo     15.8540   27343.9418  subfind_density()/subfind_density.c/93
   6 2988 0             DM_Vx    126.8318   27470.7736  subfind_density()/subfind_density.c/95
 121 2991 0    	  DensDataGet    119.5709   27845.0432  subfind_density()/subfind_density.c/161
 121 2992 0  	  DensDataResult     89.6781   27934.7213  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------   6 2989 0             DM_Vy    126.8318   27597.6054  subfind_density()/subfind_density.c/96

   6 2990 0             DM_Vz    126.8318   27724.4372  subfind_density()/subfind_density.c/97

Task=121: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.2787 MB).
task 121: endrun called with an error level of 812


   6 2991 0    	  DensDataGet    109.6591   27834.0964  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 121
   6 2992 0  	  DensDataResult     82.2443   27916.3407  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=6: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=83.6593 MB).
task 6: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 6
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   2 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   2 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   2 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   2 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   2 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   2 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
   2 2973 0         FOF_PList    380.5365   18861.3419  fof_fof()/fof.c/249
   2 2974 0         FOF_GList    634.2276   19495.5695  fof_fof()/fof.c/337
   2 2975 0             Group     11.8857   19507.4551  fof_fof()/fof.c/380
   2 2976 0   DomainNodeIndex      0.3464   19507.8015  force_treeallocate()/forcetree.c/4800
   2 2977 0        Nodes_base   3737.5142   23245.3157  force_treeallocate()/forcetree.c/4803
   2 2978 0     Extnodes_base   3114.5952   26359.9109  force_treeallocate()/forcetree.c/4811
   2 2979 0          Nextnode    222.3014   26582.2123  force_treeallocate()/forcetree.c/4820
   2 2980 0            Father    221.9056   26804.1179  force_treeallocate()/forcetree.c/4827
   2 2981 0           Ngblist     63.4228   26867.5407  subfind_density()/subfind_density.c/78
   2 2982 0         Dist2list    126.8455   26994.3862  subfind_density()/subfind_density.c/79
   2 2983 0    DataIndexTable     21.8182   27016.2044  subfind_density()/subfind_density.c/87
   2 2984 0      DataNodeList     58.1818   27074.3862  subfind_density()/subfind_density.c/89
   2 2985 0              Left    126.8455   27201.2317  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   2 2986 0             Right    126.8455   27328.0772  subfind_density()/subfind_density.c/92
   2 2987 0              Todo     15.8557   27343.9329  subfind_density()/subfind_density.c/93
   2 2988 0             DM_Vx    126.8455   27470.7784  subfind_density()/subfind_density.c/95
  34 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  34 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  34 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  34 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  34 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  34 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  34 2973 0         FOF_PList    380.1487   18860.9541  fof_fof()/fof.c/249
  34 2974 0         FOF_GList    633.5812   19494.5353  fof_fof()/fof.c/337
  34 2975 0             Group     12.3069   19506.8423  fof_fof()/fof.c/380
  34 2976 0   DomainNodeIndex      0.3464   19507.1887  force_treeallocate()/forcetree.c/4800
  34 2977 0        Nodes_base   3737.5142   23244.7029  force_treeallocate()/forcetree.c/4803
  34 2978 0     Extnodes_base   3114.5952   26359.2981  force_treeallocate()/forcetree.c/4811
  34 2979 0          Nextnode    222.3014   26581.5995  force_treeallocate()/forcetree.c/4820
   2 2989 0             DM_Vy    126.8455   27597.6239  subfind_density()/subfind_density.c/96
   2 2990 0             DM_Vz    126.8455   27724.4694  subfind_density()/subfind_density.c/97
   2 2991 0    	  DensDataGet    124.0563   27848.5257  subfind_density()/subfind_density.c/161
  34 2980 0            Father    221.9056   26803.5051  force_treeallocate()/forcetree.c/4827
  34 2981 0           Ngblist     63.3581   26866.8632  subfind_density()/subfind_density.c/78
  34 2982 0         Dist2list    126.7162   26993.5795  subfind_density()/subfind_density.c/79
  34 2983 0    DataIndexTable     21.8182   27015.3976  subfind_density()/subfind_density.c/87
   2 2992 0  	  DensDataResult     93.0422   27941.5679  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=2: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.4321 MB).
  34 2984 0      DataNodeList     58.1818   27073.5794  subfind_density()/subfind_density.c/89
  34 2985 0              Left    126.7162   27200.2957  subfind_density()/subfind_density.c/91
  34 2986 0             Right    126.7162   27327.0119  subfind_density()/subfind_density.c/92
  34 2987 0              Todo     15.8395   27342.8514  subfind_density()/subfind_density.c/93
task 2: endrun called with an error level of 812


  34 2988 0             DM_Vx    126.7162   27469.5677  subfind_density()/subfind_density.c/95
  34 2989 0             DM_Vy    126.7162   27596.2839  subfind_density()/subfind_density.c/96
  34 2990 0             DM_Vz    126.7162   27723.0002  subfind_density()/subfind_density.c/97
  34 2991 0    	  DensDataGet    122.4167   27845.4169  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 2
  34 2992 0  	  DensDataResult     91.8125   27937.2294  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=34: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.7706 MB).
task 34: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 34
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 123 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 123 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 123 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 123 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 123 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 123 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 123 2973 0         FOF_PList    380.2438   18861.0492  fof_fof()/fof.c/249
 123 2974 0         FOF_GList    633.7397   19494.7889  fof_fof()/fof.c/337
 123 2975 0             Group     11.6740   19506.4629  fof_fof()/fof.c/380
 123 2976 0   DomainNodeIndex      0.3464   19506.8093  force_treeallocate()/forcetree.c/4800
 123 2977 0        Nodes_base   3737.5142   23244.3235  force_treeallocate()/forcetree.c/4803
 123 2978 0     Extnodes_base   3114.5952   26358.9187  force_treeallocate()/forcetree.c/4811
 123 2979 0          Nextnode    222.3014   26581.2201  force_treeallocate()/forcetree.c/4820
 123 2980 0            Father    221.9056   26803.1257  force_treeallocate()/forcetree.c/4827
 123 2981 0           Ngblist     63.3740   26866.4997  subfind_density()/subfind_density.c/78
 123 2982 0         Dist2list    126.7479   26993.2476  subfind_density()/subfind_density.c/79
 123 2983 0    DataIndexTable     21.8182   27015.0658  subfind_density()/subfind_density.c/87
 123 2984 0      DataNodeList     58.1818   27073.2476  subfind_density()/subfind_density.c/89
 123 2985 0              Left    126.7479   27199.9955  subfind_density()/subfind_density.c/91
 123 2986 0             Right    126.7479   27326.7434  subfind_density()/subfind_density.c/92
 123 2987 0              Todo     15.8435   27342.5869  subfind_density()/subfind_density.c/93
 123 2988 0             DM_Vx    126.7479   27469.3349  subfind_density()/subfind_density.c/95
 123 2989 0             DM_Vy    126.7479   27596.0828  subfind_density()/subfind_density.c/96
 123 2990 0             DM_Vz    126.7479   27722.8308  subfind_density()/subfind_density.c/97
 123 2991 0    	  DensDataGet    120.6170   27843.4478  subfind_density()/subfind_density.c/161
 123 2992 0  	  DensDataResult     90.4628   27933.9105  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=123: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.0895 MB).
task 123: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 123
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  82 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  82 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  82 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  82 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  82 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  82 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  82 2973 0         FOF_PList    380.5380   18861.3433  fof_fof()/fof.c/249
  82 2974 0         FOF_GList    634.2300   19495.5733  fof_fof()/fof.c/337
  82 2975 0             Group     13.2418   19508.8151  fof_fof()/fof.c/380
  82 2976 0   DomainNodeIndex      0.3464   19509.1615  force_treeallocate()/forcetree.c/4800
  82 2977 0        Nodes_base   3737.5142   23246.6757  force_treeallocate()/forcetree.c/4803
  45 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  45 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  45 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  45 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  45 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  45 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  45 2973 0         FOF_PList    380.5399   18861.3453  fof_fof()/fof.c/249
  45 2974 0         FOF_GList    634.2332   19495.5786  fof_fof()/fof.c/337
  45 2975 0             Group     11.2849   19506.8635  fof_fof()/fof.c/380
  45 2976 0   DomainNodeIndex      0.3464   19507.2098  force_treeallocate()/forcetree.c/4800
  45 2977 0        Nodes_base   3737.5142   23244.7241  force_treeallocate()/forcetree.c/4803
  45 2978 0     Extnodes_base   3114.5952   26359.3192  force_treeallocate()/forcetree.c/4811
  45 2979 0          Nextnode    222.3014   26581.6207  force_treeallocate()/forcetree.c/4820
  45 2980 0            Father    221.9056   26803.5263  force_treeallocate()/forcetree.c/4827
  82 2978 0     Extnodes_base   3114.5952   26361.2709  force_treeallocate()/forcetree.c/4811
  82 2979 0          Nextnode    222.3014   26583.5723  force_treeallocate()/forcetree.c/4820
  82 2980 0            Father    221.9056   26805.4779  force_treeallocate()/forcetree.c/4827
  45 2981 0           Ngblist     63.4233   26866.9496  subfind_density()/subfind_density.c/78
  45 2982 0         Dist2list    126.8466   26993.7962  subfind_density()/subfind_density.c/79
  45 2983 0    DataIndexTable     21.8182   27015.6144  subfind_density()/subfind_density.c/87
  45 2984 0      DataNodeList     58.1818   27073.7962  subfind_density()/subfind_density.c/89
  82 2981 0           Ngblist     63.4230   26868.9009  subfind_density()/subfind_density.c/78
  82 2982 0         Dist2list    126.8460   26995.7469  subfind_density()/subfind_density.c/79
  82 2983 0    DataIndexTable     21.8182   27017.5651  subfind_density()/subfind_density.c/87
  45 2985 0              Left    126.8466   27200.6429  subfind_density()/subfind_density.c/91
  45 2986 0             Right    126.8466   27327.4895  subfind_density()/subfind_density.c/92
  45 2987 0              Todo     15.8558   27343.3453  subfind_density()/subfind_density.c/93
  82 2984 0      DataNodeList     58.1818   27075.7469  subfind_density()/subfind_density.c/89
  82 2985 0              Left    126.8460   27202.5929  subfind_density()/subfind_density.c/91
  82 2986 0             Right    126.8460   27329.4389  subfind_density()/subfind_density.c/92
  45 2988 0             DM_Vx    126.8466   27470.1920  subfind_density()/subfind_density.c/95
  45 2989 0             DM_Vy    126.8466   27597.0386  subfind_density()/subfind_density.c/96
  45 2990 0             DM_Vz    126.8466   27723.8853  subfind_density()/subfind_density.c/97
  82 2987 0              Todo     15.8558   27345.2946  subfind_density()/subfind_density.c/93
  82 2988 0             DM_Vx    126.8460   27472.1406  subfind_density()/subfind_density.c/95
  45 2991 0    	  DensDataGet    123.2249   27847.1101  subfind_density()/subfind_density.c/161
  45 2992 0  	  DensDataResult     92.4186   27939.5288  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=45: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.4712 MB).
  82 2989 0             DM_Vy    126.8460   27598.9866  subfind_density()/subfind_density.c/96
  82 2990 0             DM_Vz    126.8460   27725.8326  subfind_density()/subfind_density.c/97
task 45: endrun called with an error level of 812


  82 2991 0    	  DensDataGet    125.7415   27851.5741  subfind_density()/subfind_density.c/161
  82 2992 0  	  DensDataResult     94.3061   27945.8802  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 45

Task=82: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.1198 MB).
task 82: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 82
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  92 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  92 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  92 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  92 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  92 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  92 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  92 2973 0         FOF_PList    380.2099   18861.0153  fof_fof()/fof.c/249
  35 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  35 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  35 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  35 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  35 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  35 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  35 2973 0         FOF_PList    380.2928   18861.0982  fof_fof()/fof.c/249
  35 2974 0         FOF_GList    633.8214   19494.9196  fof_fof()/fof.c/337
  35 2975 0             Group     11.5661   19506.4857  fof_fof()/fof.c/380
  35 2976 0   DomainNodeIndex      0.3464   19506.8321  force_treeallocate()/forcetree.c/4800
  35 2977 0        Nodes_base   3737.5142   23244.3463  force_treeallocate()/forcetree.c/4803
  35 2978 0     Extnodes_base   3114.5952   26358.9415  force_treeallocate()/forcetree.c/4811
  35 2979 0          Nextnode    222.3014   26581.2429  force_treeallocate()/forcetree.c/4820
  92 2974 0         FOF_GList    633.6832   19494.6984  fof_fof()/fof.c/337
  92 2975 0             Group     11.6802   19506.3787  fof_fof()/fof.c/380
  92 2976 0   DomainNodeIndex      0.3464   19506.7250  force_treeallocate()/forcetree.c/4800
  35 2980 0            Father    221.9056   26803.1485  force_treeallocate()/forcetree.c/4827
  35 2981 0           Ngblist     63.3821   26866.5307  subfind_density()/subfind_density.c/78
  35 2982 0         Dist2list    126.7643   26993.2949  subfind_density()/subfind_density.c/79
  35 2983 0    DataIndexTable     21.8182   27015.1131  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  92 2977 0        Nodes_base   3737.5142   23244.2393  force_treeallocate()/forcetree.c/4803
  92 2978 0     Extnodes_base   3114.5952   26358.8344  force_treeallocate()/forcetree.c/4811
  92 2979 0          Nextnode    222.3014   26581.1359  force_treeallocate()/forcetree.c/4820
  92 2980 0            Father    221.9056   26803.0415  force_treeallocate()/forcetree.c/4827
  92 2981 0           Ngblist     63.3683   26866.4098  subfind_density()/subfind_density.c/78
  92 2982 0         Dist2list    126.7366   26993.1464  subfind_density()/subfind_density.c/79
  92 2983 0    DataIndexTable     21.8182   27014.9646  subfind_density()/subfind_density.c/87
  35 2984 0      DataNodeList     58.1818   27073.2949  subfind_density()/subfind_density.c/89
  35 2985 0              Left    126.7643   27200.0592  subfind_density()/subfind_density.c/91
  35 2986 0             Right    126.7643   27326.8235  subfind_density()/subfind_density.c/92
  35 2987 0              Todo     15.8455   27342.6690  subfind_density()/subfind_density.c/93
  33 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  33 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  33 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  33 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  33 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  33 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  33 2973 0         FOF_PList    380.4201   18861.2255  fof_fof()/fof.c/249
  33 2974 0         FOF_GList    634.0335   19495.2590  fof_fof()/fof.c/337
  33 2975 0             Group     13.0125   19508.2715  fof_fof()/fof.c/380
  33 2976 0   DomainNodeIndex      0.3464   19508.6179  force_treeallocate()/forcetree.c/4800
  33 2977 0        Nodes_base   3737.5142   23246.1321  force_treeallocate()/forcetree.c/4803
  33 2978 0     Extnodes_base   3114.5952   26360.7273  force_treeallocate()/forcetree.c/4811
  33 2979 0          Nextnode    222.3014   26583.0288  force_treeallocate()/forcetree.c/4820
  33 2980 0            Father    221.9056   26804.9343  force_treeallocate()/forcetree.c/4827
  33 2981 0           Ngblist     63.4034   26868.3377  subfind_density()/subfind_density.c/78
  92 2984 0      DataNodeList     58.1818   27073.1464  subfind_density()/subfind_density.c/89
  92 2985 0              Left    126.7366   27199.8830  subfind_density()/subfind_density.c/91
  92 2986 0             Right    126.7366   27326.6196  subfind_density()/subfind_density.c/92
  35 2988 0             DM_Vx    126.7643   27469.4333  subfind_density()/subfind_density.c/95
  35 2989 0             DM_Vy    126.7643   27596.1976  subfind_density()/subfind_density.c/96
  35 2990 0             DM_Vz    126.7643   27722.9619  subfind_density()/subfind_density.c/97
  33 2982 0         Dist2list    126.8067   26995.1444  subfind_density()/subfind_density.c/79
  33 2983 0    DataIndexTable     21.8182   27016.9626  subfind_density()/subfind_density.c/87
  33 2984 0      DataNodeList     58.1818   27075.1444  subfind_density()/subfind_density.c/89
  33 2985 0              Left    126.8067   27201.9511  subfind_density()/subfind_density.c/91
  92 2987 0              Todo     15.8421   27342.4617  subfind_density()/subfind_density.c/93
  92 2988 0             DM_Vx    126.7366   27469.1984  subfind_density()/subfind_density.c/95
  92 2989 0             DM_Vy    126.7366   27595.9350  subfind_density()/subfind_density.c/96
  35 2991 0    	  DensDataGet    109.1564   27832.1183  subfind_density()/subfind_density.c/161
  35 2992 0  	  DensDataResult     81.8673   27913.9856  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=35: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.0144 MB).
  33 2986 0             Right    126.8067   27328.7578  subfind_density()/subfind_density.c/92
  33 2987 0              Todo     15.8508   27344.6086  subfind_density()/subfind_density.c/93
  33 2988 0             DM_Vx    126.8067   27471.4154  subfind_density()/subfind_density.c/95
  92 2990 0             DM_Vz    126.7366   27722.6716  subfind_density()/subfind_density.c/97
  92 2991 0    	  DensDataGet    125.8342   27848.5059  subfind_density()/subfind_density.c/161
task 35: endrun called with an error level of 812


  33 2989 0             DM_Vy    126.8067   27598.2221  subfind_density()/subfind_density.c/96
  33 2990 0             DM_Vz    126.8067   27725.0288  subfind_density()/subfind_density.c/97
  33 2991 0    	  DensDataGet    139.1796   27864.2083  subfind_density()/subfind_density.c/161
  92 2992 0  	  DensDataResult     94.3757   27942.8815  subfind_density()/subfind_density.c/201
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 35
  33 2992 0  	  DensDataResult    104.3847   27968.5930  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------

Task=33: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=31.407 MB).
task 33: endrun called with an error level of 812



Task=92: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.1185 MB).
task 92: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 33
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 124 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 124 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 124 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 124 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 124 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 124 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 124 2973 0         FOF_PList    380.4271   18861.2325  fof_fof()/fof.c/249
 124 2974 0         FOF_GList    634.0452   19495.2776  fof_fof()/fof.c/337
 124 2975 0             Group     11.3918   19506.6694  fof_fof()/fof.c/380
 124 2976 0   DomainNodeIndex      0.3464   19507.0158  force_treeallocate()/forcetree.c/4800
 124 2977 0        Nodes_base   3737.5142   23244.5300  force_treeallocate()/forcetree.c/4803
 124 2978 0     Extnodes_base   3114.5952   26359.1252  force_treeallocate()/forcetree.c/4811
 124 2979 0          Nextnode    222.3014   26581.4266  force_treeallocate()/forcetree.c/4820
 124 2980 0            Father    221.9056   26803.3322  force_treeallocate()/forcetree.c/4827
 124 2981 0           Ngblist     63.4045   26866.7367  subfind_density()/subfind_density.c/78
 124 2982 0         Dist2list    126.8090   26993.5458  subfind_density()/subfind_density.c/79
 124 2983 0    DataIndexTable     21.8182   27015.3640  subfind_density()/subfind_density.c/87
 124 2984 0      DataNodeList     58.1818   27073.5458  subfind_density()/subfind_density.c/89
 124 2985 0              Left    126.8090   27200.3548  subfind_density()/subfind_density.c/91
 124 2986 0             Right    126.8090   27327.1638  subfind_density()/subfind_density.c/92
 124 2987 0              Todo     15.8511   27343.0150  subfind_density()/subfind_density.c/93
 124 2988 0             DM_Vx    126.8090   27469.8240  subfind_density()/subfind_density.c/95
 124 2989 0             DM_Vy    126.8090   27596.6330  subfind_density()/subfind_density.c/96
 124 2990 0             DM_Vz    126.8090   27723.4421  subfind_density()/subfind_density.c/97
 124 2991 0    	  DensDataGet    122.0957   27845.5378  subfind_density()/subfind_density.c/161
 124 2992 0  	  DensDataResult     91.5718   27937.1096  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=124: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.8904 MB).
task 124: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 124
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  83 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  83 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  83 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  83 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  83 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  83 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  83 2973 0         FOF_PList    380.4813   18861.2867  fof_fof()/fof.c/249
  83 2974 0         FOF_GList    634.1355   19495.4222  fof_fof()/fof.c/337
  83 2975 0             Group     10.7339   19506.1561  fof_fof()/fof.c/380
  83 2976 0   DomainNodeIndex      0.3464   19506.5025  force_treeallocate()/forcetree.c/4800
  83 2977 0        Nodes_base   3737.5142   23244.0167  force_treeallocate()/forcetree.c/4803
  83 2978 0     Extnodes_base   3114.5952   26358.6119  force_treeallocate()/forcetree.c/4811
  44 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  44 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  44 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  44 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  44 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  44 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  83 2979 0          Nextnode    222.3014   26580.9133  force_treeallocate()/forcetree.c/4820
  83 2980 0            Father    221.9056   26802.8189  force_treeallocate()/forcetree.c/4827
  83 2981 0           Ngblist     63.4136   26866.2325  subfind_density()/subfind_density.c/78
  83 2982 0         Dist2list    126.8271   26993.0596  subfind_density()/subfind_density.c/79
  44 2973 0         FOF_PList    380.5489   18861.3543  fof_fof()/fof.c/249
  44 2974 0         FOF_GList    634.2482   19495.6025  fof_fof()/fof.c/337
  44 2975 0             Group     13.1173   19508.7198  fof_fof()/fof.c/380
  83 2983 0    DataIndexTable     21.8182   27014.8778  subfind_density()/subfind_density.c/87
  83 2984 0      DataNodeList     58.1818   27073.0595  subfind_density()/subfind_density.c/89
  83 2985 0              Left    126.8271   27199.8867  subfind_density()/subfind_density.c/91
  83 2986 0             Right    126.8271   27326.7138  subfind_density()/subfind_density.c/92
  44 2976 0   DomainNodeIndex      0.3464   19509.0662  force_treeallocate()/forcetree.c/4800
  44 2977 0        Nodes_base   3737.5142   23246.5804  force_treeallocate()/forcetree.c/4803
  83 2987 0              Todo     15.8534   27342.5671  subfind_density()/subfind_density.c/93
  83 2988 0             DM_Vx    126.8271   27469.3942  subfind_density()/subfind_density.c/95
  83 2989 0             DM_Vy    126.8271   27596.2214  subfind_density()/subfind_density.c/96
  44 2978 0     Extnodes_base   3114.5952   26361.1756  force_treeallocate()/forcetree.c/4811
  44 2979 0          Nextnode    222.3014   26583.4770  force_treeallocate()/forcetree.c/4820
  44 2980 0            Father    221.9056   26805.3826  force_treeallocate()/forcetree.c/4827
  83 2990 0             DM_Vz    126.8271   27723.0485  subfind_density()/subfind_density.c/97
  83 2991 0    	  DensDataGet    110.2085   27833.2570  subfind_density()/subfind_density.c/161
  83 2992 0  	  DensDataResult     82.6564   27915.9133  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  44 2981 0           Ngblist     63.4248   26868.8074  subfind_density()/subfind_density.c/78
  44 2982 0         Dist2list    126.8496   26995.6570  subfind_density()/subfind_density.c/79
  44 2983 0    DataIndexTable     21.8182   27017.4752  subfind_density()/subfind_density.c/87

Task=83: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.0867 MB).
task 83: endrun called with an error level of 812


  44 2984 0      DataNodeList     58.1818   27075.6570  subfind_density()/subfind_density.c/89
  44 2985 0              Left    126.8496   27202.5067  subfind_density()/subfind_density.c/91
  44 2986 0             Right    126.8496   27329.3563  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 83
  44 2987 0              Todo     15.8562   27345.2125  subfind_density()/subfind_density.c/93
  44 2988 0             DM_Vx    126.8496   27472.0621  subfind_density()/subfind_density.c/95
  44 2989 0             DM_Vy    126.8496   27598.9118  subfind_density()/subfind_density.c/96
  44 2990 0             DM_Vz    126.8496   27725.7614  subfind_density()/subfind_density.c/97
  44 2991 0    	  DensDataGet    125.9339   27851.6953  subfind_density()/subfind_density.c/161
  44 2992 0  	  DensDataResult     94.4504   27946.1457  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=44: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=53.8543 MB).
task 44: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 44
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  40 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  40 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  40 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  40 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  40 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  40 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  40 2973 0         FOF_PList    380.0344   18860.8398  fof_fof()/fof.c/249
  40 2974 0         FOF_GList    633.3907   19494.2305  fof_fof()/fof.c/337
  40 2975 0             Group     12.1835   19506.4140  fof_fof()/fof.c/380
  40 2976 0   DomainNodeIndex      0.3464   19506.7604  force_treeallocate()/forcetree.c/4800
  40 2977 0        Nodes_base   3737.5142   23244.2746  force_treeallocate()/forcetree.c/4803
  40 2978 0     Extnodes_base   3114.5952   26358.8698  force_treeallocate()/forcetree.c/4811
  40 2979 0          Nextnode    222.3014   26581.1712  force_treeallocate()/forcetree.c/4820
  40 2980 0            Father    221.9056   26803.0768  force_treeallocate()/forcetree.c/4827
  40 2981 0           Ngblist     63.3391   26866.4159  subfind_density()/subfind_density.c/78
  40 2982 0         Dist2list    126.6781   26993.0940  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  40 2983 0    DataIndexTable     21.8182   27014.9122  subfind_density()/subfind_density.c/87
  40 2984 0      DataNodeList     58.1818   27073.0940  subfind_density()/subfind_density.c/89
 116 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 116 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 116 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 116 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 116 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 116 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 116 2973 0         FOF_PList    380.1842   18860.9896  fof_fof()/fof.c/249
 116 2974 0         FOF_GList    633.6404   19494.6299  fof_fof()/fof.c/337
 116 2975 0             Group     11.8556   19506.4855  fof_fof()/fof.c/380
 116 2976 0   DomainNodeIndex      0.3464   19506.8319  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  11 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  11 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  11 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  11 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  11 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  11 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  11 2973 0         FOF_PList    380.4388   18861.2441  fof_fof()/fof.c/249
  11 2974 0         FOF_GList    634.0646   19495.3087  fof_fof()/fof.c/337
  11 2975 0             Group     14.5129   19509.8216  fof_fof()/fof.c/380
  11 2976 0   DomainNodeIndex      0.3464   19510.1680  force_treeallocate()/forcetree.c/4800
  11 2977 0        Nodes_base   3737.5142   23247.6822  force_treeallocate()/forcetree.c/4803
  11 2978 0     Extnodes_base   3114.5952   26362.2774  force_treeallocate()/forcetree.c/4811
  11 2979 0          Nextnode    222.3014   26584.5788  force_treeallocate()/forcetree.c/4820
  11 2980 0            Father    221.9056   26806.4844  force_treeallocate()/forcetree.c/4827
  11 2981 0           Ngblist     63.4065   26869.8909  subfind_density()/subfind_density.c/78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  40 2985 0              Left    126.6781   27199.7721  subfind_density()/subfind_density.c/91
 116 2977 0        Nodes_base   3737.5142   23244.3461  force_treeallocate()/forcetree.c/4803
 116 2978 0     Extnodes_base   3114.5952   26358.9413  force_treeallocate()/forcetree.c/4811
 116 2979 0          Nextnode    222.3014   26581.2427  force_treeallocate()/forcetree.c/4820
  84 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  84 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  84 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  84 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  84 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  84 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  84 2973 0         FOF_PList    380.4862   18861.2916  fof_fof()/fof.c/249
  84 2974 0         FOF_GList    634.1437   19495.4353  fof_fof()/fof.c/337
  84 2975 0             Group     13.2263   19508.6616  fof_fof()/fof.c/380
  84 2976 0   DomainNodeIndex      0.3464   19509.0079  force_treeallocate()/forcetree.c/4800
  84 2977 0        Nodes_base   3737.5142   23246.5221  force_treeallocate()/forcetree.c/4803
  11 2982 0         Dist2list    126.8129   26996.7038  subfind_density()/subfind_density.c/79
  11 2983 0    DataIndexTable     21.8182   27018.5219  subfind_density()/subfind_density.c/87
  11 2984 0      DataNodeList     58.1818   27076.7037  subfind_density()/subfind_density.c/89
  11 2985 0              Left    126.8129   27203.5167  subfind_density()/subfind_density.c/91
  43 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  43 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  43 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  43 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  43 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  43 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  43 2973 0         FOF_PList    380.4227   18861.2281  fof_fof()/fof.c/249
  43 2974 0         FOF_GList    634.0379   19495.2659  fof_fof()/fof.c/337
  43 2975 0             Group     11.0006   19506.2665  fof_fof()/fof.c/380
  43 2976 0   DomainNodeIndex      0.3464   19506.6129  force_treeallocate()/forcetree.c/4800
  43 2977 0        Nodes_base   3737.5142   23244.1271  force_treeallocate()/forcetree.c/4803
  43 2978 0     Extnodes_base   3114.5952   26358.7223  force_treeallocate()/forcetree.c/4811
  43 2979 0          Nextnode    222.3014   26581.0238  force_treeallocate()/forcetree.c/4820
  43 2980 0            Father    221.9056   26802.9293  force_treeallocate()/forcetree.c/4827
  43 2981 0           Ngblist     63.4038   26866.3331  subfind_density()/subfind_density.c/78
  43 2982 0         Dist2list    126.8076   26993.1407  subfind_density()/subfind_density.c/79
  40 2986 0             Right    126.6781   27326.4503  subfind_density()/subfind_density.c/92
 116 2980 0            Father    221.9056   26803.1483  force_treeallocate()/forcetree.c/4827
 116 2981 0           Ngblist     63.3640   26866.5124  subfind_density()/subfind_density.c/78
 116 2982 0         Dist2list    126.7281   26993.2404  subfind_density()/subfind_density.c/79
  84 2978 0     Extnodes_base   3114.5952   26361.1173  force_treeallocate()/forcetree.c/4811
  84 2979 0          Nextnode    222.3014   26583.4188  force_treeallocate()/forcetree.c/4820
  84 2980 0            Father    221.9056   26805.3244  force_treeallocate()/forcetree.c/4827
  11 2986 0             Right    126.8129   27330.3296  subfind_density()/subfind_density.c/92
  11 2987 0              Todo     15.8516   27346.1812  subfind_density()/subfind_density.c/93
  11 2988 0             DM_Vx    126.8129   27472.9941  subfind_density()/subfind_density.c/95
  11 2989 0             DM_Vy    126.8129   27599.8070  subfind_density()/subfind_density.c/96
  43 2983 0    DataIndexTable     21.8182   27014.9589  subfind_density()/subfind_density.c/87
  43 2984 0      DataNodeList     58.1818   27073.1407  subfind_density()/subfind_density.c/89
  43 2985 0              Left    126.8076   27199.9482  subfind_density()/subfind_density.c/91
  43 2986 0             Right    126.8076   27326.7558  subfind_density()/subfind_density.c/92
  40 2987 0              Todo     15.8348   27342.2850  subfind_density()/subfind_density.c/93
 116 2983 0    DataIndexTable     21.8182   27015.0586  subfind_density()/subfind_density.c/87
 116 2984 0      DataNodeList     58.1818   27073.2404  subfind_density()/subfind_density.c/89
 116 2985 0              Left    126.7281   27199.9685  subfind_density()/subfind_density.c/91
  84 2981 0           Ngblist     63.4144   26868.7387  subfind_density()/subfind_density.c/78
  84 2982 0         Dist2list    126.8287   26995.5675  subfind_density()/subfind_density.c/79
  11 2990 0             DM_Vz    126.8129   27726.6200  subfind_density()/subfind_density.c/97
  11 2991 0    	  DensDataGet    119.3094   27845.9294  subfind_density()/subfind_density.c/161
  11 2992 0  	  DensDataResult     89.4821   27935.4115  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  43 2987 0              Todo     15.8510   27342.6068  subfind_density()/subfind_density.c/93
  43 2988 0             DM_Vx    126.8076   27469.4143  subfind_density()/subfind_density.c/95
  43 2989 0             DM_Vy    126.8076   27596.2219  subfind_density()/subfind_density.c/96
  40 2988 0             DM_Vx    126.6781   27468.9632  subfind_density()/subfind_density.c/95
 116 2986 0             Right    126.7281   27326.6966  subfind_density()/subfind_density.c/92
 116 2987 0              Todo     15.8410   27342.5376  subfind_density()/subfind_density.c/93
 116 2988 0             DM_Vx    126.7281   27469.2656  subfind_density()/subfind_density.c/95
  84 2983 0    DataIndexTable     21.8182   27017.3857  subfind_density()/subfind_density.c/87
  84 2984 0      DataNodeList     58.1818   27075.5674  subfind_density()/subfind_density.c/89
  84 2985 0              Left    126.8287   27202.3962  subfind_density()/subfind_density.c/91

Task=11: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.5885 MB).
task 11: endrun called with an error level of 812


  43 2990 0             DM_Vz    126.8076   27723.0295  subfind_density()/subfind_density.c/97
  43 2991 0    	  DensDataGet    115.1472   27838.1767  subfind_density()/subfind_density.c/161
  43 2992 0  	  DensDataResult     86.3604   27924.5371  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  40 2989 0             DM_Vy    126.6781   27595.6413  subfind_density()/subfind_density.c/96
 116 2989 0             DM_Vy    126.7281   27595.9937  subfind_density()/subfind_density.c/96
 116 2990 0             DM_Vz    126.7281   27722.7218  subfind_density()/subfind_density.c/97
  84 2986 0             Right    126.8287   27329.2249  subfind_density()/subfind_density.c/92
  84 2987 0              Todo     15.8536   27345.0785  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 11

Task=43: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.4629 MB).
task 43: endrun called with an error level of 812


  40 2990 0             DM_Vz    126.6781   27722.3195  subfind_density()/subfind_density.c/97
 116 2991 0    	  DensDataGet    121.0266   27843.7483  subfind_density()/subfind_density.c/161
 116 2992 0  	  DensDataResult     90.7699   27934.5182  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------------------------------- Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  84 2988 0             DM_Vx    126.8287   27471.9073  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 43
  40 2991 0    	  DensDataGet    134.2981   27856.6176  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

  49 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  49 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  49 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  49 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  49 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  49 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  49 2973 0         FOF_PList    378.9404   18859.7457  fof_fof()/fof.c/249
  49 2974 0         FOF_GList    631.5673   19491.3130  fof_fof()/fof.c/337
  49 2975 0             Group     12.2592   19503.5722  fof_fof()/fof.c/380
  49 2976 0   DomainNodeIndex      0.3464   19503.9186  force_treeallocate()/forcetree.c/4800
  49 2977 0        Nodes_base   3737.5142   23241.4328  force_treeallocate()/forcetree.c/4803
  49 2978 0     Extnodes_base   3114.5952   26356.0280  force_treeallocate()/forcetree.c/4811
  49 2979 0          Nextnode    222.3014   26578.3294  force_treeallocate()/forcetree.c/4820
  49 2980 0            Father    221.9056   26800.2350  force_treeallocate()/forcetree.c/4827
  84 2989 0             DM_Vy    126.8287   27598.7360  subfind_density()/subfind_density.c/96
  40 2992 0  	  DensDataResult    100.7236   27957.3412  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  78 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  78 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  78 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  78 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  78 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  78 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  78 2973 0         FOF_PList    380.5388   18861.3442  fof_fof()/fof.c/249
  78 2974 0         FOF_GList    634.2313   19495.5755  fof_fof()/fof.c/337
  78 2975 0             Group     11.8898   19507.4653  fof_fof()/fof.c/380
  78 2976 0   DomainNodeIndex      0.3464   19507.8117  force_treeallocate()/forcetree.c/4800

Task=116: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.4818 MB).
task 116: endrun called with an error level of 812


  49 2981 0           Ngblist     63.1567   26863.3917  subfind_density()/subfind_density.c/78
  49 2982 0         Dist2list    126.3135   26989.7052  subfind_density()/subfind_density.c/79
  49 2983 0    DataIndexTable     21.8182   27011.5234  subfind_density()/subfind_density.c/87
  49 2984 0      DataNodeList     58.1818   27069.7052  subfind_density()/subfind_density.c/89
  49 2985 0              Left    126.3135   27196.0186  subfind_density()/subfind_density.c/91
  84 2990 0             DM_Vz    126.8287   27725.5648  subfind_density()/subfind_density.c/97

Task=40: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.6588 MB).
  78 2977 0        Nodes_base   3737.5142   23245.3259  force_treeallocate()/forcetree.c/4803
  78 2978 0     Extnodes_base   3114.5952   26359.9211  force_treeallocate()/forcetree.c/4811
  78 2979 0          Nextnode    222.3014   26582.2225  force_treeallocate()/forcetree.c/4820
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 116
  49 2986 0             Right    126.3135   27322.3321  subfind_density()/subfind_density.c/92
  49 2987 0              Todo     15.7892   27338.1212  subfind_density()/subfind_density.c/93
  49 2988 0             DM_Vx    126.3135   27464.4347  subfind_density()/subfind_density.c/95
  84 2991 0    	  DensDataGet    132.7709   27858.3356  subfind_density()/subfind_density.c/161
task 40: endrun called with an error level of 812


  78 2980 0            Father    221.9056   26804.1281  force_treeallocate()/forcetree.c/4827
  78 2981 0           Ngblist     63.4231   26867.5513  subfind_density()/subfind_density.c/78
  78 2982 0         Dist2list    126.8463   26994.3975  subfind_density()/subfind_density.c/79
  49 2989 0             DM_Vy    126.3135   27590.7482  subfind_density()/subfind_density.c/96
  49 2990 0             DM_Vz    126.3135   27717.0616  subfind_density()/subfind_density.c/97
  49 2991 0    	  DensDataGet    116.9827   27834.0443  subfind_density()/subfind_density.c/161
  84 2992 0  	  DensDataResult     99.5782   27957.9138  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 40
  78 2983 0    DataIndexTable     21.8182   27016.2157  subfind_density()/subfind_density.c/87
  78 2984 0      DataNodeList     58.1818   27074.3975  subfind_density()/subfind_density.c/89
  78 2985 0              Left    126.8463   27201.2438  subfind_density()/subfind_density.c/91
  49 2992 0  	  DensDataResult     87.7370   27921.7813  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=49: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.2187 MB).
task 49: endrun called with an error level of 812



Task=84: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.0862 MB).
task 84: endrun called with an error level of 812


 An HDF5 error was detected, we'd better stop.
  78 2986 0             Right    126.8463   27328.0900  subfind_density()/subfind_density.c/92
  78 2987 0              Todo     15.8558   27343.9458  subfind_density()/subfind_density.c/93
  78 2988 0             DM_Vx    126.8463   27470.7921  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 49
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 84
HDF5-DIAG: Error detected in HDF5 (1.8.12)   78 2989 0             DM_Vy    126.8463   27597.6384  subfind_density()/subfind_density.c/96
MPI-process 40:
  78 2990 0             DM_Vz    126.8463   27724.4846  subfind_density()/subfind_density.c/97
task 40: endrun called with an error level of 124


  78 2991 0    	  DensDataGet    111.4860   27835.9706  subfind_density()/subfind_density.c/161
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 40
  78 2992 0  	  DensDataResult     83.6145   27919.5851  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=78: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.4149 MB).
task 78: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  48 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  48 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  48 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  48 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  48 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  48 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  86 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  86 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  86 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  86 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  86 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  86 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  86 2973 0         FOF_PList    380.1798   18860.9852  fof_fof()/fof.c/249
  86 2974 0         FOF_GList    633.6331   19494.6183  fof_fof()/fof.c/337
  86 2975 0             Group     12.7158   19507.3340  fof_fof()/fof.c/380
  41 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  41 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  41 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  41 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  41 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  41 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  41 2973 0         FOF_PList    380.3437   18861.1491  fof_fof()/fof.c/249
  41 2974 0         FOF_GList    633.9062   19495.0553  fof_fof()/fof.c/337
  41 2975 0             Group     11.9510   19507.0063  fof_fof()/fof.c/380
  41 2976 0   DomainNodeIndex      0.3464   19507.3527  force_treeallocate()/forcetree.c/4800
  41 2977 0        Nodes_base   3737.5142   23244.8669  force_treeallocate()/forcetree.c/4803
  48 2973 0         FOF_PList    380.4948   18861.3002  fof_fof()/fof.c/249
  48 2974 0         FOF_GList    634.1581   19495.4583  fof_fof()/fof.c/337
  48 2975 0             Group     12.7012   19508.1595  fof_fof()/fof.c/380
  86 2976 0   DomainNodeIndex      0.3464   19507.6804  force_treeallocate()/forcetree.c/4800
  86 2977 0        Nodes_base   3737.5142   23245.1946  force_treeallocate()/forcetree.c/4803
  86 2978 0     Extnodes_base   3114.5952   26359.7898  force_treeallocate()/forcetree.c/4811
  41 2978 0     Extnodes_base   3114.5952   26359.4621  force_treeallocate()/forcetree.c/4811
  41 2979 0          Nextnode    222.3014   26581.7636  force_treeallocate()/forcetree.c/4820
  41 2980 0            Father    221.9056   26803.6691  force_treeallocate()/forcetree.c/4827
  48 2976 0   DomainNodeIndex      0.3464   19508.5059  force_treeallocate()/forcetree.c/4800
  48 2977 0        Nodes_base   3737.5142   23246.0201  force_treeallocate()/forcetree.c/4803
  48 2978 0     Extnodes_base   3114.5952   26360.6153  force_treeallocate()/forcetree.c/4811
  86 2979 0          Nextnode    222.3014   26582.0913  force_treeallocate()/forcetree.c/4820
  86 2980 0            Father    221.9056   26803.9968  force_treeallocate()/forcetree.c/4827
  86 2981 0           Ngblist     63.3633   26867.3602  subfind_density()/subfind_density.c/78
  41 2981 0           Ngblist     63.3906   26867.0598  subfind_density()/subfind_density.c/78
  41 2982 0         Dist2list    126.7812   26993.8410  subfind_density()/subfind_density.c/79
  48 2979 0          Nextnode    222.3014   26582.9167  force_treeallocate()/forcetree.c/4820
  48 2980 0            Father    221.9056   26804.8223  force_treeallocate()/forcetree.c/4827
  86 2982 0         Dist2list    126.7266   26994.0868  subfind_density()/subfind_density.c/79
  86 2983 0    DataIndexTable     21.8182   27015.9049  subfind_density()/subfind_density.c/87
  86 2984 0      DataNodeList     58.1818   27074.0867  subfind_density()/subfind_density.c/89
  41 2983 0    DataIndexTable     21.8182   27015.6592  subfind_density()/subfind_density.c/87
  41 2984 0      DataNodeList     58.1818   27073.8410  subfind_density()/subfind_density.c/89
  48 2981 0           Ngblist     63.4158   26868.2381  subfind_density()/subfind_density.c/78
  48 2982 0         Dist2list    126.8316   26995.0697  subfind_density()/subfind_density.c/79
  48 2983 0    DataIndexTable     21.8182   27016.8879  subfind_density()/subfind_density.c/87
  86 2985 0              Left    126.7266   27200.8134  subfind_density()/subfind_density.c/91
  86 2986 0             Right    126.7266   27327.5400  subfind_density()/subfind_density.c/92
  86 2987 0              Todo     15.8408   27343.3808  subfind_density()/subfind_density.c/93
  41 2985 0              Left    126.7812   27200.6222  subfind_density()/subfind_density.c/91
  41 2986 0             Right    126.7812   27327.4035  subfind_density()/subfind_density.c/92
  41 2987 0              Todo     15.8477   27343.2511  subfind_density()/subfind_density.c/93
  48 2984 0      DataNodeList     58.1818   27075.0697  subfind_density()/subfind_density.c/89
  48 2985 0              Left    126.8316   27201.9013  subfind_density()/subfind_density.c/91
  48 2986 0             Right    126.8316   27328.7329  subfind_density()/subfind_density.c/92
  86 2988 0             DM_Vx    126.7266   27470.1074  subfind_density()/subfind_density.c/95
  86 2989 0             DM_Vy    126.7266   27596.8340  subfind_density()/subfind_density.c/96
  41 2988 0             DM_Vx    126.7812   27470.0324  subfind_density()/subfind_density.c/95
  41 2989 0             DM_Vy    126.7812   27596.8136  subfind_density()/subfind_density.c/96
  48 2987 0              Todo     15.8540   27344.5869  subfind_density()/subfind_density.c/93
  86 2990 0             DM_Vz    126.7266   27723.5606  subfind_density()/subfind_density.c/97
  86 2991 0    	  DensDataGet    125.8571   27849.4177  subfind_density()/subfind_density.c/161
  86 2992 0  	  DensDataResult     94.3928   27943.8105  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  41 2990 0             DM_Vz    126.7812   27723.5948  subfind_density()/subfind_density.c/97
  41 2991 0    	  DensDataGet    130.1339   27853.7287  subfind_density()/subfind_density.c/161
  48 2988 0             DM_Vx    126.8316   27471.4185  subfind_density()/subfind_density.c/95

Task=86: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.1895 MB).
task 86: endrun called with an error level of 812


  41 2992 0  	  DensDataResult     97.6004   27951.3291  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  48 2989 0             DM_Vy    126.8316   27598.2501  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 86

Task=41: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=48.6709 MB).
task 41: endrun called with an error level of 812


  48 2990 0             DM_Vz    126.8316   27725.0817  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 41
  48 2991 0    	  DensDataGet    114.3434   27839.4251  subfind_density()/subfind_density.c/161
  48 2992 0  	  DensDataResult     85.7575   27925.1826  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=48: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.8174 MB).
task 48: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 48
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) task 48: endrun called with an error level of 124


MPI-process 48:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 48
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  73 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  73 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  73 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  73 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  73 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  73 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  73 2973 0         FOF_PList    380.5434   18861.3488  fof_fof()/fof.c/249
  73 2974 0         FOF_GList    634.2390   19495.5878  fof_fof()/fof.c/337
  73 2975 0             Group     13.9246   19509.5123  fof_fof()/fof.c/380
  73 2976 0   DomainNodeIndex      0.3464   19509.8587  force_treeallocate()/forcetree.c/4800
  73 2977 0        Nodes_base   3737.5142   23247.3729  force_treeallocate()/forcetree.c/4803
  73 2978 0     Extnodes_base   3114.5952   26361.9681  force_treeallocate()/forcetree.c/4811
  73 2979 0          Nextnode    222.3014   26584.2695  force_treeallocate()/forcetree.c/4820
  73 2980 0            Father    221.9056   26806.1751  force_treeallocate()/forcetree.c/4827
  73 2981 0           Ngblist     63.4239   26869.5990  subfind_density()/subfind_density.c/78
  54 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  54 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  54 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  54 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  54 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  54 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  54 2973 0         FOF_PList    380.5110   18861.3163  fof_fof()/fof.c/249
  54 2974 0         FOF_GList    634.1850   19495.5013  fof_fof()/fof.c/337
  54 2975 0             Group     11.8981   19507.3994  fof_fof()/fof.c/380
  54 2976 0   DomainNodeIndex      0.3464   19507.7458  force_treeallocate()/forcetree.c/4800
  54 2977 0        Nodes_base   3737.5142   23245.2600  force_treeallocate()/forcetree.c/4803
  54 2978 0     Extnodes_base   3114.5952   26359.8552  force_treeallocate()/forcetree.c/4811
  73 2982 0         Dist2list    126.8478   26996.4468  subfind_density()/subfind_density.c/79
  73 2983 0    DataIndexTable     21.8182   27018.2650  subfind_density()/subfind_density.c/87
  73 2984 0      DataNodeList     58.1818   27076.4468  subfind_density()/subfind_density.c/89
  73 2985 0              Left    126.8478   27203.2946  subfind_density()/subfind_density.c/91
  73 2986 0             Right    126.8478   27330.1424  subfind_density()/subfind_density.c/92
  54 2979 0          Nextnode    222.3014   26582.1566  force_treeallocate()/forcetree.c/4820
  54 2980 0            Father    221.9056   26804.0622  force_treeallocate()/forcetree.c/4827
  54 2981 0           Ngblist     63.4185   26867.4807  subfind_density()/subfind_density.c/78
  73 2987 0              Todo     15.8560   27345.9984  subfind_density()/subfind_density.c/93
  73 2988 0             DM_Vx    126.8478   27472.8462  subfind_density()/subfind_density.c/95
  73 2989 0             DM_Vy    126.8478   27599.6940  subfind_density()/subfind_density.c/96
  54 2982 0         Dist2list    126.8370   26994.3177  subfind_density()/subfind_density.c/79
  54 2983 0    DataIndexTable     21.8182   27016.1359  subfind_density()/subfind_density.c/87
  54 2984 0      DataNodeList     58.1818   27074.3177  subfind_density()/subfind_density.c/89
  73 2990 0             DM_Vz    126.8478   27726.5418  subfind_density()/subfind_density.c/97
  73 2991 0    	  DensDataGet    110.4568   27836.9986  subfind_density()/subfind_density.c/161
  73 2992 0  	  DensDataResult     82.8426   27919.8412  subfind_density()/subfind_density.c/201
  54 2985 0              Left    126.8370   27201.1547  subfind_density()/subfind_density.c/91
  54 2986 0             Right    126.8370   27327.9916  subfind_density()/subfind_density.c/92
  54 2987 0              Todo     15.8546   27343.8463  subfind_density()/subfind_density.c/93
----------------------------------------------------------------------------------------

Task=73: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.1588 MB).
task 73: endrun called with an error level of 812


  54 2988 0             DM_Vx    126.8370   27470.6833  subfind_density()/subfind_density.c/95
  54 2989 0             DM_Vy    126.8370   27597.5203  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 73
  54 2990 0             DM_Vz    126.8370   27724.3572  subfind_density()/subfind_density.c/97
  54 2991 0    	  DensDataGet    114.1046   27838.4618  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  54 2992 0  	  DensDataResult     85.5784   27924.0402  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=54: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.9598 MB).
task 54: endrun called with an error level of 812


  72 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  72 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  72 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  72 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 54
  72 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  72 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  72 2973 0         FOF_PList    380.5302   18861.3355  fof_fof()/fof.c/249
  72 2974 0         FOF_GList    634.2170   19495.5525  fof_fof()/fof.c/337
  72 2975 0             Group     11.8608   19507.4133  fof_fof()/fof.c/380
  72 2976 0   DomainNodeIndex      0.3464   19507.7596  force_treeallocate()/forcetree.c/4800
  72 2977 0        Nodes_base   3737.5142   23245.2739  force_treeallocate()/forcetree.c/4803
  72 2978 0     Extnodes_base   3114.5952   26359.8690  force_treeallocate()/forcetree.c/4811
  72 2979 0          Nextnode    222.3014   26582.1705  force_treeallocate()/forcetree.c/4820
  72 2980 0            Father    221.9056   26804.0761  force_treeallocate()/forcetree.c/4827
  72 2981 0           Ngblist     63.4217   26867.4978  subfind_density()/subfind_density.c/78
  72 2982 0         Dist2list    126.8434   26994.3412  subfind_density()/subfind_density.c/79
  72 2983 0    DataIndexTable     21.8182   27016.1593  subfind_density()/subfind_density.c/87
  72 2984 0      DataNodeList     58.1818   27074.3411  subfind_density()/subfind_density.c/89
  72 2985 0              Left    126.8434   27201.1845  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  72 2986 0             Right    126.8434   27328.0279  subfind_density()/subfind_density.c/92
  63 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  63 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  63 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  63 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  63 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  63 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  63 2973 0         FOF_PList    380.4266   18861.2319  fof_fof()/fof.c/249
  63 2974 0         FOF_GList    634.0443   19495.2762  fof_fof()/fof.c/337
  63 2975 0             Group     10.7339   19506.0102  fof_fof()/fof.c/380
  63 2976 0   DomainNodeIndex      0.3464   19506.3566  force_treeallocate()/forcetree.c/4800
  63 2977 0        Nodes_base   3737.5142   23243.8708  force_treeallocate()/forcetree.c/4803
  63 2978 0     Extnodes_base   3114.5952   26358.4660  force_treeallocate()/forcetree.c/4811
  63 2979 0          Nextnode    222.3014   26580.7674  force_treeallocate()/forcetree.c/4820
  63 2980 0            Father    221.9056   26802.6730  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  72 2987 0              Todo     15.8554   27343.8833  subfind_density()/subfind_density.c/93
  63 2981 0           Ngblist     63.4044   26866.0774  subfind_density()/subfind_density.c/78
  63 2982 0         Dist2list    126.8089   26992.8863  subfind_density()/subfind_density.c/79
  63 2983 0    DataIndexTable     21.8182   27014.7045  subfind_density()/subfind_density.c/87
  63 2984 0      DataNodeList     58.1818   27072.8863  subfind_density()/subfind_density.c/89
  63 2985 0              Left    126.8089   27199.6951  subfind_density()/subfind_density.c/91
  64 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  64 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  64 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  64 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  64 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  64 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  64 2973 0         FOF_PList    380.4274   18861.2327  fof_fof()/fof.c/249
  64 2974 0         FOF_GList    634.0456   19495.2783  fof_fof()/fof.c/337
  64 2975 0             Group     12.6836   19507.9619  fof_fof()/fof.c/380
  64 2976 0   DomainNodeIndex      0.3464   19508.3083  force_treeallocate()/forcetree.c/4800
  72 2988 0             DM_Vx    126.8434   27470.7267  subfind_density()/subfind_density.c/95
  63 2986 0             Right    126.8089   27326.5040  subfind_density()/subfind_density.c/92
  63 2987 0              Todo     15.8511   27342.3551  subfind_density()/subfind_density.c/93
  63 2988 0             DM_Vx    126.8089   27469.1639  subfind_density()/subfind_density.c/95
  64 2977 0        Nodes_base   3737.5142   23245.8225  force_treeallocate()/forcetree.c/4803
  64 2978 0     Extnodes_base   3114.5952   26360.4177  force_treeallocate()/forcetree.c/4811
  64 2979 0          Nextnode    222.3014   26582.7191  force_treeallocate()/forcetree.c/4820
  64 2980 0            Father    221.9056   26804.6247  force_treeallocate()/forcetree.c/4827
  72 2989 0             DM_Vy    126.8434   27597.5701  subfind_density()/subfind_density.c/96
  63 2989 0             DM_Vy    126.8089   27595.9728  subfind_density()/subfind_density.c/96
  63 2990 0             DM_Vz    126.8089   27722.7817  subfind_density()/subfind_density.c/97
  63 2991 0    	  DensDataGet    127.9609   27850.7426  subfind_density()/subfind_density.c/161
  64 2981 0           Ngblist     63.4046   26868.0293  subfind_density()/subfind_density.c/78
  64 2982 0         Dist2list    126.8091   26994.8384  subfind_density()/subfind_density.c/79
  64 2983 0    DataIndexTable     21.8182   27016.6566  subfind_density()/subfind_density.c/87
  72 2990 0             DM_Vz    126.8434   27724.4135  subfind_density()/subfind_density.c/97
  63 2992 0  	  DensDataResult     95.9707   27946.7133  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=63: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=53.2867 MB).
task 63: endrun called with an error level of 812


  64 2984 0      DataNodeList     58.1818   27074.8384  subfind_density()/subfind_density.c/89
  64 2985 0              Left    126.8091   27201.6475  subfind_density()/subfind_density.c/91
  64 2986 0             Right    126.8091   27328.4566  subfind_density()/subfind_density.c/92
  72 2991 0    	  DensDataGet    116.7398   27841.1533  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 63
  64 2987 0              Todo     15.8511   27344.3077  subfind_density()/subfind_density.c/93
  64 2988 0             DM_Vx    126.8091   27471.1169  subfind_density()/subfind_density.c/95
  64 2989 0             DM_Vy    126.8091   27597.9260  subfind_density()/subfind_density.c/96
  72 2992 0  	  DensDataResult     87.5549   27928.7082  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  64 2990 0             DM_Vz    126.8091   27724.7351  subfind_density()/subfind_density.c/97

Task=72: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.2918 MB).
task 72: endrun called with an error level of 812


  64 2991 0    	  DensDataGet    121.5042   27846.2393  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 72
  64 2992 0  	  DensDataResult     91.1282   27937.3675  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 An HDF5 error was detected, we'd better stop.

Task=64: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.6325 MB).
task 64: endrun called with an error level of 812


HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 72------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 64
:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
  62 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  62 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  62 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  62 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  62 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
task 64: endrun called with an error level of 124


task 72: endrun called with an error level of 124


  62 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  62 2973 0         FOF_PList    379.8065   18860.6119  fof_fof()/fof.c/249
  62 2974 0         FOF_GList    633.0108   19493.6227  fof_fof()/fof.c/337
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 64:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 64
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 72
  62 2975 0             Group     12.6037   19506.2264  fof_fof()/fof.c/380
  62 2976 0   DomainNodeIndex      0.3464   19506.5727  force_treeallocate()/forcetree.c/4800
  62 2977 0        Nodes_base   3737.5142   23244.0870  force_treeallocate()/forcetree.c/4803
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  62 2978 0     Extnodes_base   3114.5952   26358.6822  force_treeallocate()/forcetree.c/4811
  62 2979 0          Nextnode    222.3014   26580.9836  force_treeallocate()/forcetree.c/4820
  62 2980 0            Father    221.9056   26802.8892  force_treeallocate()/forcetree.c/4827
  65 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  65 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  65 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  65 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  65 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  65 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  65 2973 0         FOF_PList    380.5442   18861.3496  fof_fof()/fof.c/249
  65 2974 0         FOF_GList    634.2404   19495.5900  fof_fof()/fof.c/337
  65 2975 0             Group     13.4006   19508.9905  fof_fof()/fof.c/380
  65 2976 0   DomainNodeIndex      0.3464   19509.3369  force_treeallocate()/forcetree.c/4800
  65 2977 0        Nodes_base   3737.5142   23246.8511  force_treeallocate()/forcetree.c/4803
  62 2981 0           Ngblist     63.3011   26866.1903  subfind_density()/subfind_density.c/78
  62 2982 0         Dist2list    126.6022   26992.7924  subfind_density()/subfind_density.c/79
  65 2978 0     Extnodes_base   3114.5952   26361.4463  force_treeallocate()/forcetree.c/4811
  65 2979 0          Nextnode    222.3014   26583.7477  force_treeallocate()/forcetree.c/4820
  65 2980 0            Father    221.9056   26805.6533  force_treeallocate()/forcetree.c/4827
  62 2983 0    DataIndexTable     21.8182   27014.6106  subfind_density()/subfind_density.c/87
  65 2981 0           Ngblist     63.4240   26869.0774  subfind_density()/subfind_density.c/78
  65 2982 0         Dist2list    126.8481   26995.9255  subfind_density()/subfind_density.c/79
  65 2983 0    DataIndexTable     21.8182   27017.7436  subfind_density()/subfind_density.c/87
  65 2984 0      DataNodeList     58.1818   27075.9254  subfind_density()/subfind_density.c/89
  62 2984 0      DataNodeList     58.1818   27072.7924  subfind_density()/subfind_density.c/89
  65 2985 0              Left    126.8481   27202.7735  subfind_density()/subfind_density.c/91
  65 2986 0             Right    126.8481   27329.6216  subfind_density()/subfind_density.c/92
  62 2985 0              Left    126.6022   27199.3946  subfind_density()/subfind_density.c/91
  65 2987 0              Todo     15.8560   27345.4776  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  23 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  62 2986 0             Right    126.6022   27325.9967  subfind_density()/subfind_density.c/92
  65 2988 0             DM_Vx    126.8481   27472.3257  subfind_density()/subfind_density.c/95
 104 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 104 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 104 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 104 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 104 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 104 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 104 2973 0         FOF_PList    380.5476   18861.3530  fof_fof()/fof.c/249
 104 2974 0         FOF_GList    634.2460   19495.5989  fof_fof()/fof.c/337
 104 2975 0             Group     11.1303   19506.7292  fof_fof()/fof.c/380
 104 2976 0   DomainNodeIndex      0.3464   19507.0756  force_treeallocate()/forcetree.c/4800
 104 2977 0        Nodes_base   3737.5142   23244.5898  force_treeallocate()/forcetree.c/4803
 104 2978 0     Extnodes_base   3114.5952   26359.1850  force_treeallocate()/forcetree.c/4811
 104 2979 0          Nextnode    222.3014   26581.4865  force_treeallocate()/forcetree.c/4820
  23 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  23 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  23 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  23 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  23 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  23 2973 0         FOF_PList    380.4038   18861.2092  fof_fof()/fof.c/249
  23 2974 0         FOF_GList    634.0064   19495.2156  fof_fof()/fof.c/337
  23 2975 0             Group     12.2872   19507.5028  fof_fof()/fof.c/380
  23 2976 0   DomainNodeIndex      0.3464   19507.8492  force_treeallocate()/forcetree.c/4800
  23 2977 0        Nodes_base   3737.5142   23245.3634  force_treeallocate()/forcetree.c/4803
  23 2978 0     Extnodes_base   3114.5952   26359.9586  force_treeallocate()/forcetree.c/4811
  23 2979 0          Nextnode    222.3014   26582.2600  force_treeallocate()/forcetree.c/4820
  23 2980 0            Father    221.9056   26804.1656  force_treeallocate()/forcetree.c/4827
  23 2981 0           Ngblist     63.4006   26867.5662  subfind_density()/subfind_density.c/78
  23 2982 0         Dist2list    126.8013   26994.3675  subfind_density()/subfind_density.c/79
  23 2983 0    DataIndexTable     21.8182   27016.1857  subfind_density()/subfind_density.c/87
  23 2984 0      DataNodeList     58.1818   27074.3675  subfind_density()/subfind_density.c/89
  23 2985 0              Left    126.8013   27201.1688  subfind_density()/subfind_density.c/91
  23 2986 0             Right    126.8013   27327.9700  subfind_density()/subfind_density.c/92
  23 2987 0              Todo     15.8502   27343.8202  subfind_density()/subfind_density.c/93
  23 2988 0             DM_Vx    126.8013   27470.6215  subfind_density()/subfind_density.c/95
  23 2989 0             DM_Vy    126.8013   27597.4228  subfind_density()/subfind_density.c/96
  23 2990 0             DM_Vz    126.8013   27724.2240  subfind_density()/subfind_density.c/97
  23 2991 0    	  DensDataGet    117.9580   27842.1821  subfind_density()/subfind_density.c/161
  62 2987 0              Todo     15.8253   27341.8220  subfind_density()/subfind_density.c/93
  65 2989 0             DM_Vy    126.8481   27599.1737  subfind_density()/subfind_density.c/96
 104 2980 0            Father    221.9056   26803.3920  force_treeallocate()/forcetree.c/4827
 104 2981 0           Ngblist     63.4246   26866.8166  subfind_density()/subfind_density.c/78
 104 2982 0         Dist2list    126.8492   26993.6658  subfind_density()/subfind_density.c/79
 104 2983 0    DataIndexTable     21.8182   27015.4840  subfind_density()/subfind_density.c/87
  23 2992 0  	  DensDataResult     88.4685   27930.6506  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=23: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.3494 MB).
task 23: endrun called with an error level of 812


  62 2988 0             DM_Vx    126.6022   27468.4242  subfind_density()/subfind_density.c/95
  65 2990 0             DM_Vz    126.8481   27726.0218  subfind_density()/subfind_density.c/97
 104 2984 0      DataNodeList     58.1818   27073.6658  subfind_density()/subfind_density.c/89
 104 2985 0              Left    126.8492   27200.5150  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 23
  62 2989 0             DM_Vy    126.6022   27595.0263  subfind_density()/subfind_density.c/96
  65 2991 0    	  DensDataGet    143.5153   27869.5371  subfind_density()/subfind_density.c/161
 104 2986 0             Right    126.8492   27327.3642  subfind_density()/subfind_density.c/92
 104 2987 0              Todo     15.8562   27343.2204  subfind_density()/subfind_density.c/93
 104 2988 0             DM_Vx    126.8492   27470.0696  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  62 2990 0             DM_Vz    126.6022   27721.6285  subfind_density()/subfind_density.c/97
  65 2992 0  	  DensDataResult    107.6364   27977.1735  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 104 2989 0             DM_Vy    126.8492   27596.9188  subfind_density()/subfind_density.c/96
 104 2990 0             DM_Vz    126.8492   27723.7680  subfind_density()/subfind_density.c/97
  22 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  22 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  22 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  62 2991 0    	  DensDataGet    141.3016   27862.9301  subfind_density()/subfind_density.c/161
  62 2992 0  	  DensDataResult    105.9762   27968.9062  subfind_density()/subfind_density.c/201

Task=65: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=22.8265 MB).
 104 2991 0    	  DensDataGet    114.6357   27838.4037  subfind_density()/subfind_density.c/161
 104 2992 0  	  DensDataResult     85.9768   27924.3805  subfind_density()/subfind_density.c/201
  22 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  22 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  22 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
----------------------------------------------------------------------------------------
task 65: endrun called with an error level of 812


----------------------------------------------------------------------------------------
  22 2973 0         FOF_PList    380.4916   18861.2970  fof_fof()/fof.c/249
  22 2974 0         FOF_GList    634.1527   19495.4496  fof_fof()/fof.c/337
  22 2975 0             Group     12.0475   19507.4972  fof_fof()/fof.c/380

Task=62: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=31.0938 MB).
task 62: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 65

Task=104: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.6195 MB).
  22 2976 0   DomainNodeIndex      0.3464   19507.8436  force_treeallocate()/forcetree.c/4800
  22 2977 0        Nodes_base   3737.5142   23245.3578  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 62
task 104: endrun called with an error level of 812


  22 2978 0     Extnodes_base   3114.5952   26359.9530  force_treeallocate()/forcetree.c/4811
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 104
  22 2979 0          Nextnode    222.3014   26582.2544  force_treeallocate()/forcetree.c/4820
task 104: endrun called with an error level of 124


  22 2980 0            Father    221.9056   26804.1600  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 105 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 105 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 105 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 105 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 105 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 105 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 105 2973 0         FOF_PList    380.4563   18861.2617  fof_fof()/fof.c/249
 105 2974 0         FOF_GList    634.0939   19495.3556  fof_fof()/fof.c/337
 105 2975 0             Group     11.9510   19507.3066  fof_fof()/fof.c/380
 105 2976 0   DomainNodeIndex      0.3464   19507.6530  force_treeallocate()/forcetree.c/4800
 105 2977 0        Nodes_base   3737.5142   23245.1672  force_treeallocate()/forcetree.c/4803
  22 2981 0           Ngblist     63.4153   26867.5753  subfind_density()/subfind_density.c/78
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 104:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 104
  22 2982 0         Dist2list    126.8305   26994.4058  subfind_density()/subfind_density.c/79
 105 2978 0     Extnodes_base   3114.5952   26359.7624  force_treeallocate()/forcetree.c/4811
 105 2979 0          Nextnode    222.3014   26582.0639  force_treeallocate()/forcetree.c/4820
 105 2980 0            Father    221.9056   26803.9694  force_treeallocate()/forcetree.c/4827
 105 2981 0           Ngblist     63.4094   26867.3788  subfind_density()/subfind_density.c/78
 105 2982 0         Dist2list    126.8188   26994.1976  subfind_density()/subfind_density.c/79
 105 2983 0    DataIndexTable     21.8182   27016.0158  subfind_density()/subfind_density.c/87
 105 2984 0      DataNodeList     58.1818   27074.1976  subfind_density()/subfind_density.c/89
 105 2985 0              Left    126.8188   27201.0164  subfind_density()/subfind_density.c/91
  22 2983 0    DataIndexTable     21.8182   27016.2240  subfind_density()/subfind_density.c/87
 105 2986 0             Right    126.8188   27327.8351  subfind_density()/subfind_density.c/92
 105 2987 0              Todo     15.8523   27343.6875  subfind_density()/subfind_density.c/93
 105 2988 0             DM_Vx    126.8188   27470.5063  subfind_density()/subfind_density.c/95
  22 2984 0      DataNodeList     58.1818   27074.4058  subfind_density()/subfind_density.c/89
 105 2989 0             DM_Vy    126.8188   27597.3250  subfind_density()/subfind_density.c/96
 105 2990 0             DM_Vz    126.8188   27724.1438  subfind_density()/subfind_density.c/97
 105 2991 0    	  DensDataGet    125.6613   27849.8051  subfind_density()/subfind_density.c/161
  22 2985 0              Left    126.8305   27201.2363  subfind_density()/subfind_density.c/91
 105 2992 0  	  DensDataResult     94.2460   27944.0511  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  22 2986 0             Right    126.8305   27328.0668  subfind_density()/subfind_density.c/92
  22 2987 0              Todo     15.8538   27343.9207  subfind_density()/subfind_density.c/93

Task=105: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.9489 MB).
task 105: endrun called with an error level of 812


  22 2988 0             DM_Vx    126.8305   27470.7512  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 105
  22 2989 0             DM_Vy    126.8305   27597.5817  subfind_density()/subfind_density.c/96
  22 2990 0             DM_Vz    126.8305   27724.4123  subfind_density()/subfind_density.c/97
  22 2991 0    	  DensDataGet    133.4770   27857.8893  subfind_density()/subfind_density.c/161
  22 2992 0  	  DensDataResult    100.1077   27957.9970  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=22: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.003 MB).
task 22: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 22
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  71 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  71 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  71 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  71 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  71 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  71 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  71 2973 0         FOF_PList    380.5422   18861.3476  fof_fof()/fof.c/249
  71 2974 0         FOF_GList    634.2371   19495.5847  fof_fof()/fof.c/337
  71 2975 0             Group     13.6330   19509.2177  fof_fof()/fof.c/380
  71 2976 0   DomainNodeIndex      0.3464   19509.5640  force_treeallocate()/forcetree.c/4800
  71 2977 0        Nodes_base   3737.5142   23247.0782  force_treeallocate()/forcetree.c/4803
  71 2978 0     Extnodes_base   3114.5952   26361.6734  force_treeallocate()/forcetree.c/4811
  71 2979 0          Nextnode    222.3014   26583.9749  force_treeallocate()/forcetree.c/4820
  71 2980 0            Father    221.9056   26805.8804  force_treeallocate()/forcetree.c/4827
  71 2981 0           Ngblist     63.4237   26869.3042  subfind_density()/subfind_density.c/78
  71 2982 0         Dist2list    126.8474   26996.1516  subfind_density()/subfind_density.c/79
  71 2983 0    DataIndexTable     21.8182   27017.9697  subfind_density()/subfind_density.c/87
  71 2984 0      DataNodeList     58.1818   27076.1515  subfind_density()/subfind_density.c/89
  71 2985 0              Left    126.8474   27202.9989  subfind_density()/subfind_density.c/91
  71 2986 0             Right    126.8474   27329.8464  subfind_density()/subfind_density.c/92
  71 2987 0              Todo     15.8559   27345.7023  subfind_density()/subfind_density.c/93
  71 2988 0             DM_Vx    126.8474   27472.5497  subfind_density()/subfind_density.c/95
  71 2989 0             DM_Vy    126.8474   27599.3971  subfind_density()/subfind_density.c/96
  71 2990 0             DM_Vz    126.8474   27726.2445  subfind_density()/subfind_density.c/97
  71 2991 0    	  DensDataGet    116.8881   27843.1326  subfind_density()/subfind_density.c/161
  71 2992 0  	  DensDataResult     87.6660   27930.7986  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=71: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=69.2014 MB).
task 71: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 71
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  53 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  53 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  53 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  53 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  53 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  53 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  53 2973 0         FOF_PList    380.5385   18861.3439  fof_fof()/fof.c/249
  53 2974 0         FOF_GList    634.2308   19495.5747  fof_fof()/fof.c/337
  53 2975 0             Group     11.5889   19507.1636  fof_fof()/fof.c/380
  53 2976 0   DomainNodeIndex      0.3464   19507.5100  force_treeallocate()/forcetree.c/4800
  53 2977 0        Nodes_base   3737.5142   23245.0242  force_treeallocate()/forcetree.c/4803
  53 2978 0     Extnodes_base   3114.5952   26359.6194  force_treeallocate()/forcetree.c/4811
  53 2979 0          Nextnode    222.3014   26581.9208  force_treeallocate()/forcetree.c/4820
  53 2980 0            Father    221.9056   26803.8264  force_treeallocate()/forcetree.c/4827
  53 2981 0           Ngblist     63.4231   26867.2495  subfind_density()/subfind_density.c/78
  53 2982 0         Dist2list    126.8462   26994.0957  subfind_density()/subfind_density.c/79
  53 2983 0    DataIndexTable     21.8182   27015.9139  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  53 2984 0      DataNodeList     58.1818   27074.0957  subfind_density()/subfind_density.c/89
  53 2985 0              Left    126.8462   27200.9418  subfind_density()/subfind_density.c/91
  53 2986 0             Right    126.8462   27327.7880  subfind_density()/subfind_density.c/92
  53 2987 0              Todo     15.8558   27343.6438  subfind_density()/subfind_density.c/93
  53 2988 0             DM_Vx    126.8462   27470.4899  subfind_density()/subfind_density.c/95
  53 2989 0             DM_Vy    126.8462   27597.3361  subfind_density()/subfind_density.c/96
  57 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  57 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  57 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  57 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  57 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  57 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  57 2973 0         FOF_PList    380.5318   18861.3372  fof_fof()/fof.c/249
  57 2974 0         FOF_GList    634.2197   19495.5568  fof_fof()/fof.c/337
  57 2975 0             Group     11.7124   19507.2692  fof_fof()/fof.c/380
  57 2976 0   DomainNodeIndex      0.3464   19507.6156  force_treeallocate()/forcetree.c/4800
  57 2977 0        Nodes_base   3737.5142   23245.1298  force_treeallocate()/forcetree.c/4803
  57 2978 0     Extnodes_base   3114.5952   26359.7250  force_treeallocate()/forcetree.c/4811
  57 2979 0          Nextnode    222.3014   26582.0264  force_treeallocate()/forcetree.c/4820
  53 2990 0             DM_Vz    126.8462   27724.1823  subfind_density()/subfind_density.c/97
  53 2991 0    	  DensDataGet    109.3677   27833.5499  subfind_density()/subfind_density.c/161
  53 2992 0  	  DensDataResult     82.0258   27915.5757  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  57 2980 0            Father    221.9056   26803.9320  force_treeallocate()/forcetree.c/4827
  57 2981 0           Ngblist     63.4220   26867.3540  subfind_density()/subfind_density.c/78
  57 2982 0         Dist2list    126.8439   26994.1979  subfind_density()/subfind_density.c/79
  57 2983 0    DataIndexTable     21.8182   27016.0161  subfind_density()/subfind_density.c/87
  57 2984 0      DataNodeList     58.1818   27074.1979  subfind_density()/subfind_density.c/89

Task=53: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.4243 MB).
task 53: endrun called with an error level of 812


  57 2985 0              Left    126.8439   27201.0418  subfind_density()/subfind_density.c/91
  57 2986 0             Right    126.8439   27327.8858  subfind_density()/subfind_density.c/92
  57 2987 0              Todo     15.8555   27343.7412  subfind_density()/subfind_density.c/93
  57 2988 0             DM_Vx    126.8439   27470.5852  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 53
  57 2989 0             DM_Vy    126.8439   27597.4291  subfind_density()/subfind_density.c/96
  57 2990 0             DM_Vz    126.8439   27724.2730  subfind_density()/subfind_density.c/97
  57 2991 0    	  DensDataGet    137.4818   27861.7548  subfind_density()/subfind_density.c/161
  57 2992 0  	  DensDataResult    103.1113   27964.8661  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=57: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=35.1339 MB).
task 57: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 57
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  68 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  68 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  68 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  68 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  68 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  68 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  68 2973 0         FOF_PList    380.5159   18861.3212  fof_fof()/fof.c/249
  68 2974 0         FOF_GList    634.1931   19495.5143  fof_fof()/fof.c/337
  68 2975 0             Group     10.7568   19506.2711  fof_fof()/fof.c/380
  59 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  59 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  59 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  59 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  59 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  59 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  59 2973 0         FOF_PList    380.5272   18861.3326  fof_fof()/fof.c/249
  59 2974 0         FOF_GList    634.2120   19495.5446  fof_fof()/fof.c/337
  59 2975 0             Group     11.0089   19506.5535  fof_fof()/fof.c/380
  59 2976 0   DomainNodeIndex      0.3464   19506.8998  force_treeallocate()/forcetree.c/4800
  59 2977 0        Nodes_base   3737.5142   23244.4141  force_treeallocate()/forcetree.c/4803
  59 2978 0     Extnodes_base   3114.5952   26359.0092  force_treeallocate()/forcetree.c/4811
  59 2979 0          Nextnode    222.3014   26581.3107  force_treeallocate()/forcetree.c/4820
  59 2980 0            Father    221.9056   26803.2163  force_treeallocate()/forcetree.c/4827
  59 2981 0           Ngblist     63.4212   26866.6375  subfind_density()/subfind_density.c/78
  59 2982 0         Dist2list    126.8424   26993.4799  subfind_density()/subfind_density.c/79
  59 2983 0    DataIndexTable     21.8182   27015.2980  subfind_density()/subfind_density.c/87
  59 2984 0      DataNodeList     58.1818   27073.4798  subfind_density()/subfind_density.c/89
  68 2976 0   DomainNodeIndex      0.3464   19506.6175  force_treeallocate()/forcetree.c/4800
  68 2977 0        Nodes_base   3737.5142   23244.1317  force_treeallocate()/forcetree.c/4803
  68 2978 0     Extnodes_base   3114.5952   26358.7269  force_treeallocate()/forcetree.c/4811
  59 2985 0              Left    126.8424   27200.3222  subfind_density()/subfind_density.c/91
  59 2986 0             Right    126.8424   27327.1646  subfind_density()/subfind_density.c/92
  59 2987 0              Todo     15.8553   27343.0199  subfind_density()/subfind_density.c/93
  59 2988 0             DM_Vx    126.8424   27469.8623  subfind_density()/subfind_density.c/95
  59 2989 0             DM_Vy    126.8424   27596.7047  subfind_density()/subfind_density.c/96
  59 2990 0             DM_Vz    126.8424   27723.5471  subfind_density()/subfind_density.c/97
  68 2979 0          Nextnode    222.3014   26581.0283  force_treeallocate()/forcetree.c/4820
  68 2980 0            Father    221.9056   26802.9339  force_treeallocate()/forcetree.c/4827
  68 2981 0           Ngblist     63.4193   26866.3532  subfind_density()/subfind_density.c/78
  68 2982 0         Dist2list    126.8386   26993.1918  subfind_density()/subfind_density.c/79
  59 2991 0    	  DensDataGet    117.5714   27841.1185  subfind_density()/subfind_density.c/161
  59 2992 0  	  DensDataResult     88.1786   27929.2971  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=59: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.7029 MB).
  68 2983 0    DataIndexTable     21.8182   27015.0100  subfind_density()/subfind_density.c/87
  68 2984 0      DataNodeList     58.1818   27073.1918  subfind_density()/subfind_density.c/89
  68 2985 0              Left    126.8386   27200.0304  subfind_density()/subfind_density.c/91
task 59: endrun called with an error level of 812


  68 2986 0             Right    126.8386   27326.8691  subfind_density()/subfind_density.c/92
  68 2987 0              Todo     15.8548   27342.7239  subfind_density()/subfind_density.c/93
  68 2988 0             DM_Vx    126.8386   27469.5625  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 59
  68 2989 0             DM_Vy    126.8386   27596.4011  subfind_density()/subfind_density.c/96
  68 2990 0             DM_Vz    126.8386   27723.2398  subfind_density()/subfind_density.c/97
  68 2991 0    	  DensDataGet    129.1129   27852.3527  subfind_density()/subfind_density.c/161
  68 2992 0  	  DensDataResult     96.8347   27949.1874  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=68: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.8126 MB).
task 68: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 68
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   9 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   9 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   9 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   9 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   9 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   9 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
   9 2973 0         FOF_PList    380.4171   18861.2224  fof_fof()/fof.c/249
   9 2974 0         FOF_GList    634.0285   19495.2509  fof_fof()/fof.c/337
   9 2975 0             Group     11.7446   19506.9955  fof_fof()/fof.c/380
   9 2976 0   DomainNodeIndex      0.3464   19507.3419  force_treeallocate()/forcetree.c/4800
   9 2977 0        Nodes_base   3737.5142   23244.8561  force_treeallocate()/forcetree.c/4803
   9 2978 0     Extnodes_base   3114.5952   26359.4513  force_treeallocate()/forcetree.c/4811
   9 2979 0          Nextnode    222.3014   26581.7527  force_treeallocate()/forcetree.c/4820
   9 2980 0            Father    221.9056   26803.6583  force_treeallocate()/forcetree.c/4827
   9 2981 0           Ngblist     63.4028   26867.0611  subfind_density()/subfind_density.c/78
   9 2982 0         Dist2list    126.8057   26993.8668  subfind_density()/subfind_density.c/79
   9 2983 0    DataIndexTable     21.8182   27015.6850  subfind_density()/subfind_density.c/87
   9 2984 0      DataNodeList     58.1818   27073.8668  subfind_density()/subfind_density.c/89
   9 2985 0              Left    126.8057   27200.6725  subfind_density()/subfind_density.c/91
   9 2986 0             Right    126.8057   27327.4782  subfind_density()/subfind_density.c/92
   9 2987 0              Todo     15.8507   27343.3289  subfind_density()/subfind_density.c/93
   9 2988 0             DM_Vx    126.8057   27470.1346  subfind_density()/subfind_density.c/95
   9 2989 0             DM_Vy    126.8057   27596.9403  subfind_density()/subfind_density.c/96
   9 2990 0             DM_Vz    126.8057   27723.7460  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   9 2991 0    	  DensDataGet    108.7389   27832.4849  subfind_density()/subfind_density.c/161
   9 2992 0  	  DensDataResult     81.5542   27914.0390  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  17 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  17 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  17 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  17 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  17 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  17 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  17 2973 0         FOF_PList    380.4755   18861.2809  fof_fof()/fof.c/249
  17 2974 0         FOF_GList    634.1259   19495.4067  fof_fof()/fof.c/337
  17 2975 0             Group     12.1368   19507.5435  fof_fof()/fof.c/380
  17 2976 0   DomainNodeIndex      0.3464   19507.8899  force_treeallocate()/forcetree.c/4800
  17 2977 0        Nodes_base   3737.5142   23245.4041  force_treeallocate()/forcetree.c/4803
  17 2978 0     Extnodes_base   3114.5952   26359.9993  force_treeallocate()/forcetree.c/4811
  17 2979 0          Nextnode    222.3014   26582.3007  force_treeallocate()/forcetree.c/4820
  17 2980 0            Father    221.9056   26804.2063  force_treeallocate()/forcetree.c/4827

Task=9: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.961 MB).
task 9: endrun called with an error level of 812


  17 2981 0           Ngblist     63.4126   26867.6189  subfind_density()/subfind_density.c/78
  17 2982 0         Dist2list    126.8252   26994.4441  subfind_density()/subfind_density.c/79
  17 2983 0    DataIndexTable     21.8182   27016.2623  subfind_density()/subfind_density.c/87
  17 2984 0      DataNodeList     58.1818   27074.4440  subfind_density()/subfind_density.c/89
  17 2985 0              Left    126.8252   27201.2692  subfind_density()/subfind_density.c/91
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 9
  17 2986 0             Right    126.8252   27328.0944  subfind_density()/subfind_density.c/92
  17 2987 0              Todo     15.8531   27343.9475  subfind_density()/subfind_density.c/93
  17 2988 0             DM_Vx    126.8252   27470.7727  subfind_density()/subfind_density.c/95
  17 2989 0             DM_Vy    126.8252   27597.5979  subfind_density()/subfind_density.c/96
  17 2990 0             DM_Vz    126.8252   27724.4231  subfind_density()/subfind_density.c/97
  17 2991 0    	  DensDataGet    117.1181   27841.5412  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  17 2992 0  	  DensDataResult     87.8386   27929.3797  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=17: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.6203 MB).
task 17: endrun called with an error level of 812


  67 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  67 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  67 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  67 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  67 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  67 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  67 2973 0         FOF_PList    380.5389   18861.3443  fof_fof()/fof.c/249
  67 2974 0         FOF_GList    634.2315   19495.5758  fof_fof()/fof.c/337
  67 2975 0             Group     12.1959   19507.7717  fof_fof()/fof.c/380
  67 2976 0   DomainNodeIndex      0.3464   19508.1181  force_treeallocate()/forcetree.c/4800
  67 2977 0        Nodes_base   3737.5142   23245.6323  force_treeallocate()/forcetree.c/4803
  67 2978 0     Extnodes_base   3114.5952   26360.2275  force_treeallocate()/forcetree.c/4811
  67 2979 0          Nextnode    222.3014   26582.5289  force_treeallocate()/forcetree.c/4820
  67 2980 0            Father    221.9056   26804.4345  force_treeallocate()/forcetree.c/4827
  67 2981 0           Ngblist     63.4232   26867.8577  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 17
  67 2982 0         Dist2list    126.8463   26994.7040  subfind_density()/subfind_density.c/79
  67 2983 0    DataIndexTable     21.8182   27016.5222  subfind_density()/subfind_density.c/87
  67 2984 0      DataNodeList     58.1818   27074.7040  subfind_density()/subfind_density.c/89
  67 2985 0              Left    126.8463   27201.5503  subfind_density()/subfind_density.c/91
  67 2986 0             Right    126.8463   27328.3966  subfind_density()/subfind_density.c/92
  67 2987 0              Todo     15.8558   27344.2524  subfind_density()/subfind_density.c/93
  67 2988 0             DM_Vx    126.8463   27471.0987  subfind_density()/subfind_density.c/95
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  67 2989 0             DM_Vy    126.8463   27597.9450  subfind_density()/subfind_density.c/96
  67 2990 0             DM_Vz    126.8463   27724.7913  subfind_density()/subfind_density.c/97
  67 2991 0    	  DensDataGet    109.1104   27833.9017  subfind_density()/subfind_density.c/161
  76 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  76 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  76 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  76 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  76 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  76 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  76 2973 0         FOF_PList    380.5365   18861.3419  fof_fof()/fof.c/249
  76 2974 0         FOF_GList    634.2275   19495.5694  fof_fof()/fof.c/337
  76 2975 0             Group     11.3700   19506.9394  fof_fof()/fof.c/380
  76 2976 0   DomainNodeIndex      0.3464   19507.2858  force_treeallocate()/forcetree.c/4800
  67 2992 0  	  DensDataResult     81.8328   27915.7345  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=67: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.2655 MB).
  76 2977 0        Nodes_base   3737.5142   23244.8000  force_treeallocate()/forcetree.c/4803
  76 2978 0     Extnodes_base   3114.5952   26359.3952  force_treeallocate()/forcetree.c/4811
task 67: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  76 2979 0          Nextnode    222.3014   26581.6966  force_treeallocate()/forcetree.c/4820
  76 2980 0            Father    221.9056   26803.6022  force_treeallocate()/forcetree.c/4827
  76 2981 0           Ngblist     63.4228   26867.0249  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 67
  61 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  61 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  61 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  61 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  61 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  61 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  61 2973 0         FOF_PList    380.5243   18861.3296  fof_fof()/fof.c/249
  61 2974 0         FOF_GList    634.2071   19495.5367  fof_fof()/fof.c/337
  61 2975 0             Group     10.7339   19506.2707  fof_fof()/fof.c/380
  61 2976 0   DomainNodeIndex      0.3464   19506.6171  force_treeallocate()/forcetree.c/4800
  61 2977 0        Nodes_base   3737.5142   23244.1313  force_treeallocate()/forcetree.c/4803
  61 2978 0     Extnodes_base   3114.5952   26358.7265  force_treeallocate()/forcetree.c/4811
  61 2979 0          Nextnode    222.3014   26581.0279  force_treeallocate()/forcetree.c/4820
  61 2980 0            Father    221.9056   26802.9335  force_treeallocate()/forcetree.c/4827
  61 2981 0           Ngblist     63.4207   26866.3542  subfind_density()/subfind_density.c/78
  61 2982 0         Dist2list    126.8414   26993.1956  subfind_density()/subfind_density.c/79
  61 2983 0    DataIndexTable     21.8182   27015.0138  subfind_density()/subfind_density.c/87
  42 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  42 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  42 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  42 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  42 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  76 2982 0         Dist2list    126.8455   26993.8704  subfind_density()/subfind_density.c/79
  76 2983 0    DataIndexTable     21.8182   27015.6886  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  61 2984 0      DataNodeList     58.1818   27073.1956  subfind_density()/subfind_density.c/89
  61 2985 0              Left    126.8414   27200.0370  subfind_density()/subfind_density.c/91
  61 2986 0             Right    126.8414   27326.8784  subfind_density()/subfind_density.c/92
  61 2987 0              Todo     15.8552   27342.7336  subfind_density()/subfind_density.c/93
  61 2988 0             DM_Vx    126.8414   27469.5750  subfind_density()/subfind_density.c/95
  61 2989 0             DM_Vy    126.8414   27596.4165  subfind_density()/subfind_density.c/96
  42 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  42 2973 0         FOF_PList    380.3717   18861.1771  fof_fof()/fof.c/249
  42 2974 0         FOF_GList    633.9529   19495.1299  fof_fof()/fof.c/337
  76 2984 0      DataNodeList     58.1818   27073.8704  subfind_density()/subfind_density.c/89
  76 2985 0              Left    126.8455   27200.7159  subfind_density()/subfind_density.c/91
  66 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  66 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  66 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  66 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  66 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  61 2990 0             DM_Vz    126.8414   27723.2579  subfind_density()/subfind_density.c/97
  61 2991 0    	  DensDataGet    116.3674   27839.6253  subfind_density()/subfind_density.c/161
  61 2992 0  	  DensDataResult     87.2755   27926.9008  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  42 2975 0             Group     11.7383   19506.8683  fof_fof()/fof.c/380
  42 2976 0   DomainNodeIndex      0.3464   19507.2147  force_treeallocate()/forcetree.c/4800
  42 2977 0        Nodes_base   3737.5142   23244.7289  force_treeallocate()/forcetree.c/4803
  76 2986 0             Right    126.8455   27327.5614  subfind_density()/subfind_density.c/92
  66 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  66 2973 0         FOF_PList    380.4939   18861.2992  fof_fof()/fof.c/249

Task=61: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.0992 MB).
task 61: endrun called with an error level of 812


  58 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  58 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  58 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  58 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  58 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  42 2978 0     Extnodes_base   3114.5952   26359.3241  force_treeallocate()/forcetree.c/4811
  42 2979 0          Nextnode    222.3014   26581.6255  force_treeallocate()/forcetree.c/4820
  42 2980 0            Father    221.9056   26803.5311  force_treeallocate()/forcetree.c/4827
  76 2987 0              Todo     15.8557   27343.4171  subfind_density()/subfind_density.c/93
  66 2974 0         FOF_GList    634.1565   19495.4557  fof_fof()/fof.c/337
  66 2975 0             Group     11.7996   19507.2552  fof_fof()/fof.c/380
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 61
  58 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  58 2973 0         FOF_PList    380.5325   18861.3379  fof_fof()/fof.c/249
  42 2981 0           Ngblist     63.3953   26866.9264  subfind_density()/subfind_density.c/78
  42 2982 0         Dist2list    126.7906   26993.7169  subfind_density()/subfind_density.c/79
  42 2983 0    DataIndexTable     21.8182   27015.5351  subfind_density()/subfind_density.c/87
  76 2988 0             DM_Vx    126.8455   27470.2626  subfind_density()/subfind_density.c/95
  66 2976 0   DomainNodeIndex      0.3464   19507.6016  force_treeallocate()/forcetree.c/4800
  66 2977 0        Nodes_base   3737.5142   23245.1158  force_treeallocate()/forcetree.c/4803
  58 2974 0         FOF_GList    634.2209   19495.5588  fof_fof()/fof.c/337
  58 2975 0             Group     12.7905   19508.3492  fof_fof()/fof.c/380
  58 2976 0   DomainNodeIndex      0.3464   19508.6956  force_treeallocate()/forcetree.c/4800
  42 2984 0      DataNodeList     58.1818   27073.7169  subfind_density()/subfind_density.c/89
  42 2985 0              Left    126.7906   27200.5075  subfind_density()/subfind_density.c/91
  76 2989 0             DM_Vy    126.8455   27597.1081  subfind_density()/subfind_density.c/96
  66 2978 0     Extnodes_base   3114.5952   26359.7110  force_treeallocate()/forcetree.c/4811
  66 2979 0          Nextnode    222.3014   26582.0125  force_treeallocate()/forcetree.c/4820
  58 2977 0        Nodes_base   3737.5142   23246.2098  force_treeallocate()/forcetree.c/4803
  58 2978 0     Extnodes_base   3114.5952   26360.8050  force_treeallocate()/forcetree.c/4811
  58 2979 0          Nextnode    222.3014   26583.1065  force_treeallocate()/forcetree.c/4820
  42 2986 0             Right    126.7906   27327.2981  subfind_density()/subfind_density.c/92
  76 2990 0             DM_Vz    126.8455   27723.9536  subfind_density()/subfind_density.c/97
  66 2980 0            Father    221.9056   26803.9180  force_treeallocate()/forcetree.c/4827
  66 2981 0           Ngblist     63.4156   26867.3337  subfind_density()/subfind_density.c/78
  58 2980 0            Father    221.9056   26805.0120  force_treeallocate()/forcetree.c/4827
  58 2981 0           Ngblist     63.4221   26868.4341  subfind_density()/subfind_density.c/78
  58 2982 0         Dist2list    126.8442   26995.2783  subfind_density()/subfind_density.c/79
  42 2987 0              Todo     15.8488   27343.1469  subfind_density()/subfind_density.c/93
  76 2991 0    	  DensDataGet    138.0775   27862.0311  subfind_density()/subfind_density.c/161
  66 2982 0         Dist2list    126.8313   26994.1650  subfind_density()/subfind_density.c/79
  58 2983 0    DataIndexTable     21.8182   27017.0965  subfind_density()/subfind_density.c/87
  58 2984 0      DataNodeList     58.1818   27075.2783  subfind_density()/subfind_density.c/89
  42 2988 0             DM_Vx    126.7906   27469.9375  subfind_density()/subfind_density.c/95
  76 2992 0  	  DensDataResult    103.5581   27965.5892  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  66 2983 0    DataIndexTable     21.8182   27015.9832  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  58 2985 0              Left    126.8442   27202.1225  subfind_density()/subfind_density.c/91
  58 2986 0             Right    126.8442   27328.9666  subfind_density()/subfind_density.c/92
  42 2989 0             DM_Vy    126.7906   27596.7280  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=76: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=34.4108 MB).
task 76: endrun called with an error level of 812


  66 2984 0      DataNodeList     58.1818   27074.1650  subfind_density()/subfind_density.c/89
  81 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  81 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  81 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  81 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  81 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  81 2973 0         FOF_PList    380.5336   18861.3390  fof_fof()/fof.c/249
  81 2974 0         FOF_GList    634.2227   19495.5616  fof_fof()/fof.c/337
  81 2975 0             Group     10.7339   19506.2956  fof_fof()/fof.c/380
  81 2976 0   DomainNodeIndex      0.3464   19506.6420  force_treeallocate()/forcetree.c/4800
  81 2977 0        Nodes_base   3737.5142   23244.1562  force_treeallocate()/forcetree.c/4803
  81 2978 0     Extnodes_base   3114.5952   26358.7514  force_treeallocate()/forcetree.c/4811
  81 2979 0          Nextnode    222.3014   26581.0528  force_treeallocate()/forcetree.c/4820
  81 2980 0            Father    221.9056   26802.9584  force_treeallocate()/forcetree.c/4827
  81 2981 0           Ngblist     63.4223   26866.3807  subfind_density()/subfind_density.c/78
 119 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 119 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 119 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 119 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 119 2973 0         FOF_PList    380.5155   18861.3209  fof_fof()/fof.c/249
 119 2974 0         FOF_GList    634.1926   19495.5135  fof_fof()/fof.c/337
 119 2975 0             Group     11.2569   19506.7704  fof_fof()/fof.c/380
 119 2976 0   DomainNodeIndex      0.3464   19507.1168  force_treeallocate()/forcetree.c/4800
 119 2977 0        Nodes_base   3737.5142   23244.6310  force_treeallocate()/forcetree.c/4803
 119 2978 0     Extnodes_base   3114.5952   26359.2262  force_treeallocate()/forcetree.c/4811
 119 2979 0          Nextnode    222.3014   26581.5276  force_treeallocate()/forcetree.c/4820
 119 2980 0            Father    221.9056   26803.4332  force_treeallocate()/forcetree.c/4827
 119 2981 0           Ngblist     63.4193   26866.8524  subfind_density()/subfind_density.c/78
  58 2987 0              Todo     15.8555   27344.8222  subfind_density()/subfind_density.c/93
  42 2990 0             DM_Vz    126.7906   27723.5186  subfind_density()/subfind_density.c/97
  25 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  25 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  25 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  25 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  25 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  25 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  25 2973 0         FOF_PList    380.4689   18861.2742  fof_fof()/fof.c/249
  25 2974 0         FOF_GList    634.1148   19495.3890  fof_fof()/fof.c/337
  25 2975 0             Group     11.5474   19506.9365  fof_fof()/fof.c/380
  25 2976 0   DomainNodeIndex      0.3464   19507.2828  force_treeallocate()/forcetree.c/4800
  25 2977 0        Nodes_base   3737.5142   23244.7971  force_treeallocate()/forcetree.c/4803
 102 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 102 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 102 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 102 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 102 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 102 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 102 2973 0         FOF_PList    380.5386   18861.3439  fof_fof()/fof.c/249
 102 2974 0         FOF_GList    634.2310   19495.5749  fof_fof()/fof.c/337
 102 2975 0             Group     12.0351   19507.6100  fof_fof()/fof.c/380
  46 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  46 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  46 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  46 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  46 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  46 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  46 2973 0         FOF_PList    380.5038   18861.3092  fof_fof()/fof.c/249
  46 2974 0         FOF_GList    634.1730   19495.4822  fof_fof()/fof.c/337
  46 2975 0             Group     11.0234   19506.5057  fof_fof()/fof.c/380
  46 2976 0   DomainNodeIndex      0.3464   19506.8520  force_treeallocate()/forcetree.c/4800
   8 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   8 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   8 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   8 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   8 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   8 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 76
  66 2985 0              Left    126.8313   27200.9962  subfind_density()/subfind_density.c/91
  81 2982 0         Dist2list    126.8445   26993.2252  subfind_density()/subfind_density.c/79
  81 2983 0    DataIndexTable     21.8182   27015.0434  subfind_density()/subfind_density.c/87
  81 2984 0      DataNodeList     58.1818   27073.2252  subfind_density()/subfind_density.c/89
  81 2985 0              Left    126.8445   27200.0697  subfind_density()/subfind_density.c/91
 119 2982 0         Dist2list    126.8385   26993.6910  subfind_density()/subfind_density.c/79
 119 2983 0    DataIndexTable     21.8182   27015.5091  subfind_density()/subfind_density.c/87
 119 2984 0      DataNodeList     58.1818   27073.6909  subfind_density()/subfind_density.c/89
 119 2985 0              Left    126.8385   27200.5294  subfind_density()/subfind_density.c/91
  58 2988 0             DM_Vx    126.8442   27471.6663  subfind_density()/subfind_density.c/95
  42 2991 0    	  DensDataGet    113.4658   27836.9844  subfind_density()/subfind_density.c/161
  25 2978 0     Extnodes_base   3114.5952   26359.3922  force_treeallocate()/forcetree.c/4811
  25 2979 0          Nextnode    222.3014   26581.6937  force_treeallocate()/forcetree.c/4820
  25 2980 0            Father    221.9056   26803.5993  force_treeallocate()/forcetree.c/4827
  25 2981 0           Ngblist     63.4115   26867.0107  subfind_density()/subfind_density.c/78
 102 2976 0   DomainNodeIndex      0.3464   19507.9564  force_treeallocate()/forcetree.c/4800
 102 2977 0        Nodes_base   3737.5142   23245.4706  force_treeallocate()/forcetree.c/4803
 102 2978 0     Extnodes_base   3114.5952   26360.0658  force_treeallocate()/forcetree.c/4811
  46 2977 0        Nodes_base   3737.5142   23244.3663  force_treeallocate()/forcetree.c/4803
  46 2978 0     Extnodes_base   3114.5952   26358.9614  force_treeallocate()/forcetree.c/4811
  46 2979 0          Nextnode    222.3014   26581.2629  force_treeallocate()/forcetree.c/4820
   8 2973 0         FOF_PList    380.3686   18861.1740  fof_fof()/fof.c/249
   8 2974 0         FOF_GList    633.9477   19495.1217  fof_fof()/fof.c/337
   8 2975 0             Group     13.2159   19508.3376  fof_fof()/fof.c/380
  66 2986 0             Right    126.8313   27327.8275  subfind_density()/subfind_density.c/92
  81 2986 0             Right    126.8445   27326.9142  subfind_density()/subfind_density.c/92
  81 2987 0              Todo     15.8556   27342.7698  subfind_density()/subfind_density.c/93
  81 2988 0             DM_Vx    126.8445   27469.6143  subfind_density()/subfind_density.c/95
 119 2986 0             Right    126.8385   27327.3680  subfind_density()/subfind_density.c/92
 119 2987 0              Todo     15.8548   27343.2228  subfind_density()/subfind_density.c/93
 119 2988 0             DM_Vx    126.8385   27470.0613  subfind_density()/subfind_density.c/95
  58 2989 0             DM_Vy    126.8442   27598.5105  subfind_density()/subfind_density.c/96
  42 2992 0  	  DensDataResult     85.0993   27922.0837  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  25 2982 0         Dist2list    126.8230   26993.8337  subfind_density()/subfind_density.c/79
  25 2983 0    DataIndexTable     21.8182   27015.6519  subfind_density()/subfind_density.c/87
 102 2979 0          Nextnode    222.3014   26582.3672  force_treeallocate()/forcetree.c/4820
 102 2980 0            Father    221.9056   26804.2728  force_treeallocate()/forcetree.c/4827
 102 2981 0           Ngblist     63.4231   26867.6959  subfind_density()/subfind_density.c/78
  46 2980 0            Father    221.9056   26803.1685  force_treeallocate()/forcetree.c/4827
  46 2981 0           Ngblist     63.4173   26866.5858  subfind_density()/subfind_density.c/78
  46 2982 0         Dist2list    126.8346   26993.4204  subfind_density()/subfind_density.c/79
  46 2983 0    DataIndexTable     21.8182   27015.2386  subfind_density()/subfind_density.c/87
   8 2976 0   DomainNodeIndex      0.3464   19508.6840  force_treeallocate()/forcetree.c/4800
   8 2977 0        Nodes_base   3737.5142   23246.1982  force_treeallocate()/forcetree.c/4803
  66 2987 0              Todo     15.8539   27343.6814  subfind_density()/subfind_density.c/93
  81 2989 0             DM_Vy    126.8445   27596.4589  subfind_density()/subfind_density.c/96
  81 2990 0             DM_Vz    126.8445   27723.3034  subfind_density()/subfind_density.c/97
  81 2991 0    	  DensDataGet    116.0598   27839.3632  subfind_density()/subfind_density.c/161
 119 2989 0             DM_Vy    126.8385   27596.8998  subfind_density()/subfind_density.c/96
 119 2990 0             DM_Vz    126.8385   27723.7383  subfind_density()/subfind_density.c/97
 119 2991 0    	  DensDataGet    122.7378   27846.4761  subfind_density()/subfind_density.c/161
  58 2990 0             DM_Vz    126.8442   27725.3547  subfind_density()/subfind_density.c/97

  25 2984 0      DataNodeList     58.1818   27073.8337  subfind_density()/subfind_density.c/89
  25 2985 0              Left    126.8230   27200.6566  subfind_density()/subfind_density.c/91
  25 2986 0             Right    126.8230   27327.4796  subfind_density()/subfind_density.c/92
 102 2982 0         Dist2list    126.8462   26994.5421  subfind_density()/subfind_density.c/79
 102 2983 0    DataIndexTable     21.8182   27016.3602  subfind_density()/subfind_density.c/87
 102 2984 0      DataNodeList     58.1818   27074.5420  subfind_density()/subfind_density.c/89
  46 2984 0      DataNodeList     58.1818   27073.4204  subfind_density()/subfind_density.c/89
  46 2985 0              Left    126.8346   27200.2550  subfind_density()/subfind_density.c/91
  46 2986 0             Right    126.8346   27327.0896  subfind_density()/subfind_density.c/92
   8 2978 0     Extnodes_base   3114.5952   26360.7934  force_treeallocate()/forcetree.c/4811
   8 2979 0          Nextnode    222.3014   26583.0948  force_treeallocate()/forcetree.c/4820
  66 2988 0             DM_Vx    126.8313   27470.5127  subfind_density()/subfind_density.c/95
  81 2992 0  	  DensDataResult     87.0449   27926.4081  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=81: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.5919 MB).
task 81: endrun called with an error level of 812


 119 2992 0  	  DensDataResult     92.0533   27938.5295  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=119: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.4705 MB).
task 119: endrun called with an error level of 812


  58 2991 0    	  DensDataGet    142.9936   27868.3483  subfind_density()/subfind_density.c/161

Task=42: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.9163 MB).
task 42: endrun called with an error level of 812


  25 2987 0              Todo     15.8529   27343.3325  subfind_density()/subfind_density.c/93
  25 2988 0             DM_Vx    126.8230   27470.1554  subfind_density()/subfind_density.c/95
 102 2985 0              Left    126.8462   27201.3882  subfind_density()/subfind_density.c/91
 102 2986 0             Right    126.8462   27328.2344  subfind_density()/subfind_density.c/92
 102 2987 0              Todo     15.8558   27344.0902  subfind_density()/subfind_density.c/93
  46 2987 0              Todo     15.8543   27342.9439  subfind_density()/subfind_density.c/93
  46 2988 0             DM_Vx    126.8346   27469.7785  subfind_density()/subfind_density.c/95
  46 2989 0             DM_Vy    126.8346   27596.6131  subfind_density()/subfind_density.c/96
   8 2980 0            Father    221.9056   26805.0004  force_treeallocate()/forcetree.c/4827
   8 2981 0           Ngblist     63.3948   26868.3952  subfind_density()/subfind_density.c/78
   8 2982 0         Dist2list    126.7895   26995.1847  subfind_density()/subfind_density.c/79
  66 2989 0             DM_Vy    126.8313   27597.3440  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 81
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 119
  58 2992 0  	  DensDataResult    107.2452   27975.5935  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 42
  25 2989 0             DM_Vy    126.8230   27596.9784  subfind_density()/subfind_density.c/96
  25 2990 0             DM_Vz    126.8230   27723.8013  subfind_density()/subfind_density.c/97
 102 2988 0             DM_Vx    126.8462   27470.9364  subfind_density()/subfind_density.c/95
  46 2990 0             DM_Vz    126.8346   27723.4477  subfind_density()/subfind_density.c/97
   8 2983 0    DataIndexTable     21.8182   27017.0029  subfind_density()/subfind_density.c/87
   8 2984 0      DataNodeList     58.1818   27075.1847  subfind_density()/subfind_density.c/89
  66 2990 0             DM_Vz    126.8313   27724.1753  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=58: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=24.4065 MB).
task 58: endrun called with an error level of 812


  25 2991 0    	  DensDataGet    115.0662   27838.8675  subfind_density()/subfind_density.c/161
  25 2992 0  	  DensDataResult     86.2996   27925.1671  subfind_density()/subfind_density.c/201
 102 2989 0             DM_Vy    126.8462   27597.7826  subfind_density()/subfind_density.c/96
  46 2991 0    	  DensDataGet    136.7505   27860.1983  subfind_density()/subfind_density.c/161
   8 2985 0              Left    126.7895   27201.9742  subfind_density()/subfind_density.c/91
   8 2986 0             Right    126.7895   27328.7638  subfind_density()/subfind_density.c/92
  66 2991 0    	  DensDataGet    129.0059   27853.1812  subfind_density()/subfind_density.c/161
  80 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  80 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  80 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  80 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  80 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  80 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  80 2973 0         FOF_PList    380.5391   18861.3445  fof_fof()/fof.c/249
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 58
----------------------------------------------------------------------------------------
 102 2990 0             DM_Vz    126.8462   27724.6288  subfind_density()/subfind_density.c/97
 102 2991 0    	  DensDataGet    125.5925   27850.2212  subfind_density()/subfind_density.c/161
 102 2992 0  	  DensDataResult     94.1944   27944.4156  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  46 2992 0  	  DensDataResult    102.5629   27962.7612  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   8 2987 0              Todo     15.8487   27344.6124  subfind_density()/subfind_density.c/93
   8 2988 0             DM_Vx    126.7895   27471.4020  subfind_density()/subfind_density.c/95
   8 2989 0             DM_Vy    126.7895   27598.1915  subfind_density()/subfind_density.c/96
   8 2990 0             DM_Vz    126.7895   27724.9811  subfind_density()/subfind_density.c/97
  66 2992 0  	  DensDataResult     96.7544   27949.9357  subfind_density()/subfind_density.c/201
  80 2974 0         FOF_GList    634.2319   19495.5764  fof_fof()/fof.c/337
  80 2975 0             Group     12.5674   19508.1437  fof_fof()/fof.c/380

Task=25: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.8329 MB).
task 25: endrun called with an error level of 812



Task=102: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.5844 MB).
task 102: endrun called with an error level of 812



Task=46: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=37.2388 MB).
task 46: endrun called with an error level of 812


   8 2991 0    	  DensDataGet    111.1114   27836.0925  subfind_density()/subfind_density.c/161
----------------------------------------------------------------------------------------
  80 2976 0   DomainNodeIndex      0.3464   19508.4901  force_treeallocate()/forcetree.c/4800
  80 2977 0        Nodes_base   3737.5142   23246.0043  force_treeallocate()/forcetree.c/4803
  80 2978 0     Extnodes_base   3114.5952   26360.5995  force_treeallocate()/forcetree.c/4811
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 25
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 102
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 46
   8 2992 0  	  DensDataResult     83.3335   27919.4260  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=8: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.574 MB).
task 8: endrun called with an error level of 812



Task=66: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.0643 MB).
  80 2979 0          Nextnode    222.3014   26582.9009  force_treeallocate()/forcetree.c/4820
  80 2980 0            Father    221.9056   26804.8065  force_treeallocate()/forcetree.c/4827
  80 2981 0           Ngblist     63.4232   26868.2297  subfind_density()/subfind_density.c/78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 8
 An HDF5 error was detected, we'd better stop.
task 66: endrun called with an error level of 812


  80 2982 0         Dist2list    126.8464   26995.0761  subfind_density()/subfind_density.c/79
  80 2983 0    DataIndexTable     21.8182   27016.8943  subfind_density()/subfind_density.c/87
  47 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  47 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  47 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  47 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  47 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  47 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  47 2973 0         FOF_PList    380.5364   18861.3418  fof_fof()/fof.c/249
  47 2974 0         FOF_GList    634.2274   19495.5692  fof_fof()/fof.c/337
  47 2975 0             Group     12.3744   19507.9436  fof_fof()/fof.c/380
  47 2976 0   DomainNodeIndex      0.3464   19508.2900  force_treeallocate()/forcetree.c/4800
  47 2977 0        Nodes_base   3737.5142   23245.8042  force_treeallocate()/forcetree.c/4803
task 8: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 66
  80 2984 0      DataNodeList     58.1818   27075.0761  subfind_density()/subfind_density.c/89
  80 2985 0              Left    126.8464   27201.9224  subfind_density()/subfind_density.c/91
  80 2986 0             Right    126.8464   27328.7688  subfind_density()/subfind_density.c/92
  47 2978 0     Extnodes_base   3114.5952   26360.3994  force_treeallocate()/forcetree.c/4811
  47 2979 0          Nextnode    222.3014   26582.7008  force_treeallocate()/forcetree.c/4820
  47 2980 0            Father    221.9056   26804.6064  force_treeallocate()/forcetree.c/4827
  47 2981 0           Ngblist     63.4227   26868.0291  subfind_density()/subfind_density.c/78
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 8:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 8
  80 2987 0              Todo     15.8558   27344.6246  subfind_density()/subfind_density.c/93
  47 2982 0         Dist2list    126.8455   26994.8746  subfind_density()/subfind_density.c/79
  47 2983 0    DataIndexTable     21.8182   27016.6928  subfind_density()/subfind_density.c/87
  47 2984 0      DataNodeList     58.1818   27074.8746  subfind_density()/subfind_density.c/89
  80 2988 0             DM_Vx    126.8464   27471.4710  subfind_density()/subfind_density.c/95
  47 2985 0              Left    126.8455   27201.7201  subfind_density()/subfind_density.c/91
  47 2986 0             Right    126.8455   27328.5656  subfind_density()/subfind_density.c/92
  47 2987 0              Todo     15.8557   27344.4212  subfind_density()/subfind_density.c/93
  47 2988 0             DM_Vx    126.8455   27471.2667  subfind_density()/subfind_density.c/95
  80 2989 0             DM_Vy    126.8464   27598.3174  subfind_density()/subfind_density.c/96
  47 2989 0             DM_Vy    126.8455   27598.1122  subfind_density()/subfind_density.c/96
  47 2990 0             DM_Vz    126.8455   27724.9577  subfind_density()/subfind_density.c/97
  47 2991 0    	  DensDataGet    111.7779   27836.7356  subfind_density()/subfind_density.c/161
  80 2990 0             DM_Vz    126.8464   27725.1637  subfind_density()/subfind_density.c/97
  47 2992 0  	  DensDataResult     83.8334   27920.5690  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=47: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.431 MB).
task 47: endrun called with an error level of 812


  80 2991 0    	  DensDataGet    135.9572   27861.1209  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 47
  80 2992 0  	  DensDataResult    101.9679   27963.0889  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=80: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=36.9111 MB).
task 80: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 80
  14 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  14 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  14 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  14 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  14 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  14 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  14 2973 0         FOF_PList    380.5358   18861.3412  fof_fof()/fof.c/249
  14 2974 0         FOF_GList    634.2264   19495.5676  fof_fof()/fof.c/337
  14 2975 0             Group     10.7339   19506.3015  fof_fof()/fof.c/380
  14 2976 0   DomainNodeIndex      0.3464   19506.6479  force_treeallocate()/forcetree.c/4800
  14 2977 0        Nodes_base   3737.5142   23244.1621  force_treeallocate()/forcetree.c/4803
 113 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 113 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 113 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 113 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 113 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 113 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 113 2973 0         FOF_PList    380.4673   18861.2727  fof_fof()/fof.c/249
 113 2974 0         FOF_GList    634.1122   19495.3848  fof_fof()/fof.c/337
 113 2975 0             Group     12.3692   19507.7540  fof_fof()/fof.c/380
 113 2976 0   DomainNodeIndex      0.3464   19508.1004  force_treeallocate()/forcetree.c/4800
 113 2977 0        Nodes_base   3737.5142   23245.6146  force_treeallocate()/forcetree.c/4803
 An HDF5 error was detected, we'd better stop.
  14 2978 0     Extnodes_base   3114.5952   26358.7573  force_treeallocate()/forcetree.c/4811
  14 2979 0          Nextnode    222.3014   26581.0587  force_treeallocate()/forcetree.c/4820
  14 2980 0            Father    221.9056   26802.9643  force_treeallocate()/forcetree.c/4827
 113 2978 0     Extnodes_base   3114.5952   26360.2098  force_treeallocate()/forcetree.c/4811
 113 2979 0          Nextnode    222.3014   26582.5112  force_treeallocate()/forcetree.c/4820
 113 2980 0            Father    221.9056   26804.4168  force_treeallocate()/forcetree.c/4827
 113 2981 0           Ngblist     63.4112   26867.8280  subfind_density()/subfind_density.c/78
HDF5-DIAG: Error detected in HDF5 (1.8.12)   14 2981 0           Ngblist     63.4226   26866.3869  subfind_density()/subfind_density.c/78
  14 2982 0         Dist2list    126.8453   26993.2322  subfind_density()/subfind_density.c/79
  14 2983 0    DataIndexTable     21.8182   27015.0504  subfind_density()/subfind_density.c/87
 113 2982 0         Dist2list    126.8224   26994.6505  subfind_density()/subfind_density.c/79
 113 2983 0    DataIndexTable     21.8182   27016.4687  subfind_density()/subfind_density.c/87
 113 2984 0      DataNodeList     58.1818   27074.6504  subfind_density()/subfind_density.c/89
MPI-process 80:
  14 2984 0      DataNodeList     58.1818   27073.2322  subfind_density()/subfind_density.c/89
  14 2985 0              Left    126.8453   27200.0775  subfind_density()/subfind_density.c/91
  14 2986 0             Right    126.8453   27326.9227  subfind_density()/subfind_density.c/92
 113 2985 0              Left    126.8224   27201.4729  subfind_density()/subfind_density.c/91
 113 2986 0             Right    126.8224   27328.2953  subfind_density()/subfind_density.c/92
 113 2987 0              Todo     15.8528   27344.1481  subfind_density()/subfind_density.c/93
task 80: endrun called with an error level of 124


  14 2987 0              Todo     15.8557   27342.7784  subfind_density()/subfind_density.c/93
  14 2988 0             DM_Vx    126.8453   27469.6237  subfind_density()/subfind_density.c/95
  14 2989 0             DM_Vy    126.8453   27596.4690  subfind_density()/subfind_density.c/96
 113 2988 0             DM_Vx    126.8224   27470.9706  subfind_density()/subfind_density.c/95
 113 2989 0             DM_Vy    126.8224   27597.7930  subfind_density()/subfind_density.c/96
 113 2990 0             DM_Vz    126.8224   27724.6154  subfind_density()/subfind_density.c/97
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 80
  14 2990 0             DM_Vz    126.8453   27723.3142  subfind_density()/subfind_density.c/97
 113 2991 0    	  DensDataGet    114.9922   27839.6076  subfind_density()/subfind_density.c/161
  14 2991 0    	  DensDataGet    115.9361   27839.2503  subfind_density()/subfind_density.c/161
 113 2992 0  	  DensDataResult     86.2441   27925.8517  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  14 2992 0  	  DensDataResult     86.9521   27926.2024  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=113: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.1483 MB).
task 113: endrun called with an error level of 812



Task=14: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.7976 MB).
task 14: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 113
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 14
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 107 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 107 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 107 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 107 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 107 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 107 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 107 2973 0         FOF_PList    380.2801   18861.0854  fof_fof()/fof.c/249
 107 2974 0         FOF_GList    633.8001   19494.8856  fof_fof()/fof.c/337
 107 2975 0             Group     13.2792   19508.1647  fof_fof()/fof.c/380
 107 2976 0   DomainNodeIndex      0.3464   19508.5111  force_treeallocate()/forcetree.c/4800
 107 2977 0        Nodes_base   3737.5142   23246.0253  force_treeallocate()/forcetree.c/4803
 107 2978 0     Extnodes_base   3114.5952   26360.6205  force_treeallocate()/forcetree.c/4811
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 107 2979 0          Nextnode    222.3014   26582.9219  force_treeallocate()/forcetree.c/4820
 107 2980 0            Father    221.9056   26804.8275  force_treeallocate()/forcetree.c/4827
 107 2981 0           Ngblist     63.3800   26868.2075  subfind_density()/subfind_density.c/78
 107 2982 0         Dist2list    126.7600   26994.9676  subfind_density()/subfind_density.c/79
  26 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  26 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  26 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  26 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  26 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  26 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  26 2973 0         FOF_PList    380.3489   18861.1542  fof_fof()/fof.c/249
  26 2974 0         FOF_GList    633.9148   19495.0690  fof_fof()/fof.c/337
  26 2975 0             Group     11.6387   19506.7077  fof_fof()/fof.c/380
  26 2976 0   DomainNodeIndex      0.3464   19507.0541  force_treeallocate()/forcetree.c/4800
 101 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 101 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 101 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 101 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 101 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 101 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 101 2973 0         FOF_PList    380.5311   18861.3364  fof_fof()/fof.c/249
 101 2974 0         FOF_GList    634.2184   19495.5549  fof_fof()/fof.c/337
 101 2975 0             Group     10.7339   19506.2888  fof_fof()/fof.c/380
 101 2976 0   DomainNodeIndex      0.3464   19506.6352  force_treeallocate()/forcetree.c/4800
 101 2977 0        Nodes_base   3737.5142   23244.1494  force_treeallocate()/forcetree.c/4803
 101 2978 0     Extnodes_base   3114.5952   26358.7446  force_treeallocate()/forcetree.c/4811
 101 2979 0          Nextnode    222.3014   26581.0460  force_treeallocate()/forcetree.c/4820
 101 2980 0            Father    221.9056   26802.9516  force_treeallocate()/forcetree.c/4827
 101 2981 0           Ngblist     63.4218   26866.3735  subfind_density()/subfind_density.c/78
 107 2983 0    DataIndexTable     21.8182   27016.7857  subfind_density()/subfind_density.c/87
 107 2984 0      DataNodeList     58.1818   27074.9675  subfind_density()/subfind_density.c/89
 107 2985 0              Left    126.7600   27201.7276  subfind_density()/subfind_density.c/91
 107 2986 0             Right    126.7600   27328.4876  subfind_density()/subfind_density.c/92
  26 2977 0        Nodes_base   3737.5142   23244.5683  force_treeallocate()/forcetree.c/4803
  26 2978 0     Extnodes_base   3114.5952   26359.1635  force_treeallocate()/forcetree.c/4811
  26 2979 0          Nextnode    222.3014   26581.4649  force_treeallocate()/forcetree.c/4820
  26 2980 0            Father    221.9056   26803.3705  force_treeallocate()/forcetree.c/4827
 101 2982 0         Dist2list    126.8437   26993.2171  subfind_density()/subfind_density.c/79
 101 2983 0    DataIndexTable     21.8182   27015.0353  subfind_density()/subfind_density.c/87
 101 2984 0      DataNodeList     58.1818   27073.2171  subfind_density()/subfind_density.c/89
 101 2985 0              Left    126.8437   27200.0608  subfind_density()/subfind_density.c/91
 101 2986 0             Right    126.8437   27326.9045  subfind_density()/subfind_density.c/92
 107 2987 0              Todo     15.8450   27344.3326  subfind_density()/subfind_density.c/93
 107 2988 0             DM_Vx    126.7600   27471.0926  subfind_density()/subfind_density.c/95
  26 2981 0           Ngblist     63.3915   26866.7620  subfind_density()/subfind_density.c/78
  26 2982 0         Dist2list    126.7830   26993.5449  subfind_density()/subfind_density.c/79
  26 2983 0    DataIndexTable     21.8182   27015.3631  subfind_density()/subfind_density.c/87
 101 2987 0              Todo     15.8555   27342.7600  subfind_density()/subfind_density.c/93
 101 2988 0             DM_Vx    126.8437   27469.6036  subfind_density()/subfind_density.c/95
 101 2989 0             DM_Vy    126.8437   27596.4473  subfind_density()/subfind_density.c/96
 107 2989 0             DM_Vy    126.7600   27597.8526  subfind_density()/subfind_density.c/96
 107 2990 0             DM_Vz    126.7600   27724.6127  subfind_density()/subfind_density.c/97
 107 2991 0    	  DensDataGet    111.1448   27835.7575  subfind_density()/subfind_density.c/161
  26 2984 0      DataNodeList     58.1818   27073.5449  subfind_density()/subfind_density.c/89
  26 2985 0              Left    126.7830   27200.3279  subfind_density()/subfind_density.c/91
  26 2986 0             Right    126.7830   27327.1108  subfind_density()/subfind_density.c/92
 101 2990 0             DM_Vz    126.8437   27723.2910  subfind_density()/subfind_density.c/97
 101 2991 0    	  DensDataGet    125.3126   27848.6036  subfind_density()/subfind_density.c/161
 101 2992 0  	  DensDataResult     93.9845   27942.5881  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 107 2992 0  	  DensDataResult     83.3586   27919.1161  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  26 2987 0              Todo     15.8479   27342.9587  subfind_density()/subfind_density.c/93
  26 2988 0             DM_Vx    126.7830   27469.7416  subfind_density()/subfind_density.c/95
  26 2989 0             DM_Vy    126.7830   27596.5246  subfind_density()/subfind_density.c/96

Task=101: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.4119 MB).
task 101: endrun called with an error level of 812



Task=107: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=80.8839 MB).
  26 2990 0             DM_Vz    126.7830   27723.3075  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 101
task 107: endrun called with an error level of 812


  26 2991 0    	  DensDataGet    109.6488   27832.9563  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 107
  26 2992 0  	  DensDataResult     82.2366   27915.1929  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=26: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.8071 MB).
task 26: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 26
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 111 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 111 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 111 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 111 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 111 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 111 2973 0         FOF_PList    380.5245   18861.3298  fof_fof()/fof.c/249
 111 2974 0         FOF_GList    634.2074   19495.5372  fof_fof()/fof.c/337
 111 2975 0             Group     12.3817   19507.9189  fof_fof()/fof.c/380
 111 2976 0   DomainNodeIndex      0.3464   19508.2653  force_treeallocate()/forcetree.c/4800
 111 2977 0        Nodes_base   3737.5142   23245.7795  force_treeallocate()/forcetree.c/4803
 111 2978 0     Extnodes_base   3114.5952   26360.3747  force_treeallocate()/forcetree.c/4811
 111 2979 0          Nextnode    222.3014   26582.6761  force_treeallocate()/forcetree.c/4820
 111 2980 0            Father    221.9056   26804.5817  force_treeallocate()/forcetree.c/4827
 111 2981 0           Ngblist     63.4207   26868.0024  subfind_density()/subfind_density.c/78
 111 2982 0         Dist2list    126.8415   26994.8439  subfind_density()/subfind_density.c/79
 111 2983 0    DataIndexTable     21.8182   27016.6621  subfind_density()/subfind_density.c/87
 111 2984 0      DataNodeList     58.1818   27074.8439  subfind_density()/subfind_density.c/89
 111 2985 0              Left    126.8415   27201.6854  subfind_density()/subfind_density.c/91
 111 2986 0             Right    126.8415   27328.5268  subfind_density()/subfind_density.c/92
  16 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  16 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  16 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  16 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  16 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  16 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  16 2973 0         FOF_PList    380.3049   18861.1103  fof_fof()/fof.c/249
 111 2987 0              Todo     15.8552   27344.3820  subfind_density()/subfind_density.c/93
 111 2988 0             DM_Vx    126.8415   27471.2235  subfind_density()/subfind_density.c/95
 111 2989 0             DM_Vy    126.8415   27598.0650  subfind_density()/subfind_density.c/96
 111 2990 0             DM_Vz    126.8415   27724.9065  subfind_density()/subfind_density.c/97
 111 2991 0    	  DensDataGet    116.0151   27840.9216  subfind_density()/subfind_density.c/161
 111 2992 0  	  DensDataResult     87.0113   27927.9329  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  16 2974 0         FOF_GList    633.8416   19494.9518  fof_fof()/fof.c/337
  16 2975 0             Group     12.3817   19507.3335  fof_fof()/fof.c/380
  16 2976 0   DomainNodeIndex      0.3464   19507.6799  force_treeallocate()/forcetree.c/4800
  16 2977 0        Nodes_base   3737.5142   23245.1941  force_treeallocate()/forcetree.c/4803

Task=111: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.0671 MB).
task 111: endrun called with an error level of 812


  16 2978 0     Extnodes_base   3114.5952   26359.7893  force_treeallocate()/forcetree.c/4811
  16 2979 0          Nextnode    222.3014   26582.0907  force_treeallocate()/forcetree.c/4820
  16 2980 0            Father    221.9056   26803.9963  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 111
  16 2981 0           Ngblist     63.3842   26867.3804  subfind_density()/subfind_density.c/78
  16 2982 0         Dist2list    126.7683   26994.1488  subfind_density()/subfind_density.c/79
  16 2983 0    DataIndexTable     21.8182   27015.9669  subfind_density()/subfind_density.c/87
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2984 0      DataNodeList     58.1818   27074.1487  subfind_density()/subfind_density.c/89
  16 2985 0              Left    126.7683   27200.9170  subfind_density()/subfind_density.c/91
 106 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 106 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 106 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 106 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 106 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 106 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 106 2973 0         FOF_PList    380.2045   18861.0099  fof_fof()/fof.c/249
 106 2974 0         FOF_GList    633.6742   19494.6841  fof_fof()/fof.c/337
 106 2975 0             Group     11.9127   19506.5968  fof_fof()/fof.c/380
  16 2986 0             Right    126.7683   27327.6853  subfind_density()/subfind_density.c/92
  16 2987 0              Todo     15.8460   27343.5314  subfind_density()/subfind_density.c/93
  16 2988 0             DM_Vx    126.7683   27470.2997  subfind_density()/subfind_density.c/95
 106 2976 0   DomainNodeIndex      0.3464   19506.9432  force_treeallocate()/forcetree.c/4800
 106 2977 0        Nodes_base   3737.5142   23244.4574  force_treeallocate()/forcetree.c/4803
 106 2978 0     Extnodes_base   3114.5952   26359.0526  force_treeallocate()/forcetree.c/4811
 106 2979 0          Nextnode    222.3014   26581.3540  force_treeallocate()/forcetree.c/4820
  16 2989 0             DM_Vy    126.7683   27597.0680  subfind_density()/subfind_density.c/96
 106 2980 0            Father    221.9056   26803.2596  force_treeallocate()/forcetree.c/4827
 106 2981 0           Ngblist     63.3674   26866.6270  subfind_density()/subfind_density.c/78
 106 2982 0         Dist2list    126.7348   26993.3619  subfind_density()/subfind_density.c/79
 106 2983 0    DataIndexTable     21.8182   27015.1800  subfind_density()/subfind_density.c/87
  16 2990 0             DM_Vz    126.7683   27723.8363  subfind_density()/subfind_density.c/97
 106 2984 0      DataNodeList     58.1818   27073.3618  subfind_density()/subfind_density.c/89
 106 2985 0              Left    126.7348   27200.0967  subfind_density()/subfind_density.c/91
 106 2986 0             Right    126.7348   27326.8315  subfind_density()/subfind_density.c/92
 106 2987 0              Todo     15.8419   27342.6734  subfind_density()/subfind_density.c/93
 106 2988 0             DM_Vx    126.7348   27469.4082  subfind_density()/subfind_density.c/95
  16 2991 0    	  DensDataGet    118.5192   27842.3555  subfind_density()/subfind_density.c/161
 106 2989 0             DM_Vy    126.7348   27596.1431  subfind_density()/subfind_density.c/96
 106 2990 0             DM_Vz    126.7348   27722.8779  subfind_density()/subfind_density.c/97
 106 2991 0    	  DensDataGet    150.6246   27873.5025  subfind_density()/subfind_density.c/161
  16 2992 0  	  DensDataResult     88.8894   27931.2449  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 106 2992 0  	  DensDataResult    112.9684   27986.4709  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=106: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=13.5291 MB).
task 106: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=16: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.7551 MB).
task 16: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 106
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 114 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 114 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 114 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 114 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 114 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 114 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 114 2973 0         FOF_PList    380.5358   18861.3411  fof_fof()/fof.c/249
 114 2974 0         FOF_GList    634.2263   19495.5674  fof_fof()/fof.c/337
 114 2975 0             Group     11.3617   19506.9291  fof_fof()/fof.c/380
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 16
  13 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  13 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  13 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  13 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  13 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  13 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  13 2973 0         FOF_PList    380.4152   18861.2206  fof_fof()/fof.c/249
  13 2974 0         FOF_GList    634.0253   19495.2459  fof_fof()/fof.c/337
  13 2975 0             Group     12.1617   19507.4076  fof_fof()/fof.c/380
  13 2976 0   DomainNodeIndex      0.3464   19507.7540  force_treeallocate()/forcetree.c/4800
  13 2977 0        Nodes_base   3737.5142   23245.2682  force_treeallocate()/forcetree.c/4803
  13 2978 0     Extnodes_base   3114.5952   26359.8634  force_treeallocate()/forcetree.c/4811
  13 2979 0          Nextnode    222.3014   26582.1648  force_treeallocate()/forcetree.c/4820
  13 2980 0            Father    221.9056   26804.0704  force_treeallocate()/forcetree.c/4827
  13 2981 0           Ngblist     63.4025   26867.4729  subfind_density()/subfind_density.c/78
  13 2982 0         Dist2list    126.8051   26994.2780  subfind_density()/subfind_density.c/79
  13 2983 0    DataIndexTable     21.8182   27016.0962  subfind_density()/subfind_density.c/87
  13 2984 0      DataNodeList     58.1818   27074.2780  subfind_density()/subfind_density.c/89
  13 2985 0              Left    126.8051   27201.0830  subfind_density()/subfind_density.c/91
  13 2986 0             Right    126.8051   27327.8881  subfind_density()/subfind_density.c/92
  13 2987 0              Todo     15.8506   27343.7387  subfind_density()/subfind_density.c/93
 114 2976 0   DomainNodeIndex      0.3464   19507.2754  force_treeallocate()/forcetree.c/4800
 114 2977 0        Nodes_base   3737.5142   23244.7897  force_treeallocate()/forcetree.c/4803
 114 2978 0     Extnodes_base   3114.5952   26359.3848  force_treeallocate()/forcetree.c/4811
 114 2979 0          Nextnode    222.3014   26581.6863  force_treeallocate()/forcetree.c/4820
 An HDF5 error was detected, we'd better stop.
  13 2988 0             DM_Vx    126.8051   27470.5438  subfind_density()/subfind_density.c/95
  13 2989 0             DM_Vy    126.8051   27597.3489  subfind_density()/subfind_density.c/96
  13 2990 0             DM_Vz    126.8051   27724.1539  subfind_density()/subfind_density.c/97
  13 2991 0    	  DensDataGet    134.4380   27858.5920  subfind_density()/subfind_density.c/161
  13 2992 0  	  DensDataResult    100.8285   27959.4205  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 114 2980 0            Father    221.9056   26803.5919  force_treeallocate()/forcetree.c/4827
 114 2981 0           Ngblist     63.4226   26867.0145  subfind_density()/subfind_density.c/78
 114 2982 0         Dist2list    126.8453   26993.8598  subfind_density()/subfind_density.c/79
HDF5-DIAG: Error detected in HDF5 (1.8.12) 
Task=13: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=40.5795 MB).
task 13: endrun called with an error level of 812


 114 2983 0    DataIndexTable     21.8182   27015.6779  subfind_density()/subfind_density.c/87
 114 2984 0      DataNodeList     58.1818   27073.8597  subfind_density()/subfind_density.c/89
 114 2985 0              Left    126.8453   27200.7050  subfind_density()/subfind_density.c/91
MPI-process 16:
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 13
 114 2986 0             Right    126.8453   27327.5502  subfind_density()/subfind_density.c/92
 114 2987 0              Todo     15.8557   27343.4059  subfind_density()/subfind_density.c/93
 114 2988 0             DM_Vx    126.8453   27470.2511  subfind_density()/subfind_density.c/95
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
 114 2989 0             DM_Vy    126.8453   27597.0964  subfind_density()/subfind_density.c/96
task 16: endrun called with an error level of 124


 114 2990 0             DM_Vz    126.8453   27723.9417  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 16
 114 2991 0    	  DensDataGet    127.9966   27851.9383  subfind_density()/subfind_density.c/161
 114 2992 0  	  DensDataResult     95.9975   27947.9358  subfind_density()/subfind_density.c/201
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
  24 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  24 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  24 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  24 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  24 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  12 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  12 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  12 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  12 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  12 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  12 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  12 2973 0         FOF_PList    380.5389   18861.3443  fof_fof()/fof.c/249
  12 2974 0         FOF_GList    634.2315   19495.5758  fof_fof()/fof.c/337
  12 2975 0             Group     11.9926   19507.5684  fof_fof()/fof.c/380

Task=114: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.0642 MB).
  24 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  24 2973 0         FOF_PList    380.5195   18861.3249  fof_fof()/fof.c/249
  12 2976 0   DomainNodeIndex      0.3464   19507.9147  force_treeallocate()/forcetree.c/4800
  12 2977 0        Nodes_base   3737.5142   23245.4289  force_treeallocate()/forcetree.c/4803
  12 2978 0     Extnodes_base   3114.5952   26360.0241  force_treeallocate()/forcetree.c/4811
task 114: endrun called with an error level of 812


  24 2974 0         FOF_GList    634.1992   19495.5241  fof_fof()/fof.c/337
  24 2975 0             Group     11.9282   19507.4523  fof_fof()/fof.c/380
  12 2979 0          Nextnode    222.3014   26582.3256  force_treeallocate()/forcetree.c/4820
  12 2980 0            Father    221.9056   26804.2312  force_treeallocate()/forcetree.c/4827
  12 2981 0           Ngblist     63.4232   26867.6543  subfind_density()/subfind_density.c/78
  12 2982 0         Dist2list    126.8463   26994.5006  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 114
  24 2976 0   DomainNodeIndex      0.3464   19507.7987  force_treeallocate()/forcetree.c/4800
  24 2977 0        Nodes_base   3737.5142   23245.3129  force_treeallocate()/forcetree.c/4803
  12 2983 0    DataIndexTable     21.8182   27016.3188  subfind_density()/subfind_density.c/87
  12 2984 0      DataNodeList     58.1818   27074.5006  subfind_density()/subfind_density.c/89
  12 2985 0              Left    126.8463   27201.3469  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  24 2978 0     Extnodes_base   3114.5952   26359.9081  force_treeallocate()/forcetree.c/4811
  24 2979 0          Nextnode    222.3014   26582.2095  force_treeallocate()/forcetree.c/4820
  12 2986 0             Right    126.8463   27328.1932  subfind_density()/subfind_density.c/92
  12 2987 0              Todo     15.8558   27344.0490  subfind_density()/subfind_density.c/93
  12 2988 0             DM_Vx    126.8463   27470.8953  subfind_density()/subfind_density.c/95
 115 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 115 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 115 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 115 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 115 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 115 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 115 2973 0         FOF_PList    380.4433   18861.2486  fof_fof()/fof.c/249
 115 2974 0         FOF_GList    634.0721   19495.3207  fof_fof()/fof.c/337
 115 2975 0             Group     11.2527   19506.5735  fof_fof()/fof.c/380
 115 2976 0   DomainNodeIndex      0.3464   19506.9199  force_treeallocate()/forcetree.c/4800
 115 2977 0        Nodes_base   3737.5142   23244.4341  force_treeallocate()/forcetree.c/4803
 115 2978 0     Extnodes_base   3114.5952   26359.0293  force_treeallocate()/forcetree.c/4811
  24 2980 0            Father    221.9056   26804.1151  force_treeallocate()/forcetree.c/4827
  24 2981 0           Ngblist     63.4199   26867.5350  subfind_density()/subfind_density.c/78
  12 2989 0             DM_Vy    126.8463   27597.7416  subfind_density()/subfind_density.c/96
  12 2990 0             DM_Vz    126.8463   27724.5879  subfind_density()/subfind_density.c/97
 115 2979 0          Nextnode    222.3014   26581.3307  force_treeallocate()/forcetree.c/4820
 115 2980 0            Father    221.9056   26803.2363  force_treeallocate()/forcetree.c/4827
 115 2981 0           Ngblist     63.4072   26866.6435  subfind_density()/subfind_density.c/78
  24 2982 0         Dist2list    126.8398   26994.3749  subfind_density()/subfind_density.c/79
  24 2983 0    DataIndexTable     21.8182   27016.1931  subfind_density()/subfind_density.c/87
  12 2991 0    	  DensDataGet    107.8765   27832.4644  subfind_density()/subfind_density.c/161
 115 2982 0         Dist2list    126.8144   26993.4579  subfind_density()/subfind_density.c/79
 115 2983 0    DataIndexTable     21.8182   27015.2761  subfind_density()/subfind_density.c/87
 115 2984 0      DataNodeList     58.1818   27073.4579  subfind_density()/subfind_density.c/89
 115 2985 0              Left    126.8144   27200.2723  subfind_density()/subfind_density.c/91
  24 2984 0      DataNodeList     58.1818   27074.3749  subfind_density()/subfind_density.c/89
  12 2992 0  	  DensDataResult     80.9073   27913.3717  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 115 2986 0             Right    126.8144   27327.0867  subfind_density()/subfind_density.c/92
 115 2987 0              Todo     15.8518   27342.9385  subfind_density()/subfind_density.c/93
 115 2988 0             DM_Vx    126.8144   27469.7530  subfind_density()/subfind_density.c/95
  24 2985 0              Left    126.8398   27201.2147  subfind_density()/subfind_density.c/91

Task=12: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.6283 MB).
task 12: endrun called with an error level of 812


 115 2989 0             DM_Vy    126.8144   27596.5674  subfind_density()/subfind_density.c/96
 115 2990 0             DM_Vz    126.8144   27723.3818  subfind_density()/subfind_density.c/97
 115 2991 0    	  DensDataGet    115.1760   27838.5578  subfind_density()/subfind_density.c/161
 115 2992 0  	  DensDataResult     86.3820   27924.9398  subfind_density()/subfind_density.c/201
  24 2986 0             Right    126.8398   27328.0546  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 12
----------------------------------------------------------------------------------------

Task=115: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.0602 MB).
task 115: endrun called with an error level of 812


  24 2987 0              Todo     15.8550   27343.9095  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 115
  24 2988 0             DM_Vx    126.8398   27470.7494  subfind_density()/subfind_density.c/95
  24 2989 0             DM_Vy    126.8398   27597.5892  subfind_density()/subfind_density.c/96
  24 2990 0             DM_Vz    126.8398   27724.4291  subfind_density()/subfind_density.c/97
  24 2991 0    	  DensDataGet    108.4590   27832.8880  subfind_density()/subfind_density.c/161
  24 2992 0  	  DensDataResult     81.3442   27914.2323  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=24: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.7677 MB).
task 24: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 24
task 24: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 24:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 24
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  15 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  15 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  15 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  15 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  15 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  15 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  15 2973 0         FOF_PList    380.2564   18861.0617  fof_fof()/fof.c/249
  15 2974 0         FOF_GList    633.7606   19494.8224  fof_fof()/fof.c/337
  15 2975 0             Group     12.4491   19507.2715  fof_fof()/fof.c/380
  15 2976 0   DomainNodeIndex      0.3464   19507.6179  force_treeallocate()/forcetree.c/4800
  15 2977 0        Nodes_base   3737.5142   23245.1321  force_treeallocate()/forcetree.c/4803
  15 2978 0     Extnodes_base   3114.5952   26359.7273  force_treeallocate()/forcetree.c/4811
  15 2979 0          Nextnode    222.3014   26582.0287  force_treeallocate()/forcetree.c/4820
  15 2980 0            Father    221.9056   26803.9343  force_treeallocate()/forcetree.c/4827
 112 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 112 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 112 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 112 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 112 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 112 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 112 2973 0         FOF_PList    380.5266   18861.3319  fof_fof()/fof.c/249
  27 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  27 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  27 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  27 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  27 2973 0         FOF_PList    380.4815   18861.2869  fof_fof()/fof.c/249
  27 2974 0         FOF_GList    634.1359   19495.4227  fof_fof()/fof.c/337
  27 2975 0             Group     12.5020   19507.9247  fof_fof()/fof.c/380
  27 2976 0   DomainNodeIndex      0.3464   19508.2711  force_treeallocate()/forcetree.c/4800
  27 2977 0        Nodes_base   3737.5142   23245.7853  force_treeallocate()/forcetree.c/4803
  27 2978 0     Extnodes_base   3114.5952   26360.3805  force_treeallocate()/forcetree.c/4811
 100 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 100 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 100 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 100 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 100 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 100 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  15 2981 0           Ngblist     63.3761   26867.3103  subfind_density()/subfind_density.c/78
  15 2982 0         Dist2list    126.7521   26994.0625  subfind_density()/subfind_density.c/79
  15 2983 0    DataIndexTable     21.8182   27015.8807  subfind_density()/subfind_density.c/87
  15 2984 0      DataNodeList     58.1818   27074.0624  subfind_density()/subfind_density.c/89
 112 2974 0         FOF_GList    634.2109   19495.5428  fof_fof()/fof.c/337
 112 2975 0             Group     13.4182   19508.9611  fof_fof()/fof.c/380
 112 2976 0   DomainNodeIndex      0.3464   19509.3074  force_treeallocate()/forcetree.c/4800
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  27 2979 0          Nextnode    222.3014   26582.6820  force_treeallocate()/forcetree.c/4820
  27 2980 0            Father    221.9056   26804.5875  force_treeallocate()/forcetree.c/4827
  27 2981 0           Ngblist     63.4136   26868.0011  subfind_density()/subfind_density.c/78
  27 2982 0         Dist2list    126.8272   26994.8283  subfind_density()/subfind_density.c/79
 100 2973 0         FOF_PList    380.5139   18861.3193  fof_fof()/fof.c/249
 100 2974 0         FOF_GList    634.1899   19495.5092  fof_fof()/fof.c/337
 100 2975 0             Group     12.1025   19507.6118  fof_fof()/fof.c/380
  15 2985 0              Left    126.7521   27200.8146  subfind_density()/subfind_density.c/91
  15 2986 0             Right    126.7521   27327.5667  subfind_density()/subfind_density.c/92
  15 2987 0              Todo     15.8440   27343.4107  subfind_density()/subfind_density.c/93
  15 2988 0             DM_Vx    126.7521   27470.1628  subfind_density()/subfind_density.c/95
 112 2977 0        Nodes_base   3737.5142   23246.8216  force_treeallocate()/forcetree.c/4803
 112 2978 0     Extnodes_base   3114.5952   26361.4168  force_treeallocate()/forcetree.c/4811
 112 2979 0          Nextnode    222.3014   26583.7183  force_treeallocate()/forcetree.c/4820
  98 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  98 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  98 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  98 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  98 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  98 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  98 2973 0         FOF_PList    380.5247   18861.3300  fof_fof()/fof.c/249
  98 2974 0         FOF_GList    634.2078   19495.5378  fof_fof()/fof.c/337
  98 2975 0             Group     12.1067   19507.6445  fof_fof()/fof.c/380
  98 2976 0   DomainNodeIndex      0.3464   19507.9908  force_treeallocate()/forcetree.c/4800
  98 2977 0        Nodes_base   3737.5142   23245.5051  force_treeallocate()/forcetree.c/4803
  98 2978 0     Extnodes_base   3114.5952   26360.1002  force_treeallocate()/forcetree.c/4811
  98 2979 0          Nextnode    222.3014   26582.4017  force_treeallocate()/forcetree.c/4820
  27 2983 0    DataIndexTable     21.8182   27016.6465  subfind_density()/subfind_density.c/87
  27 2984 0      DataNodeList     58.1818   27074.8283  subfind_density()/subfind_density.c/89
  27 2985 0              Left    126.8272   27201.6554  subfind_density()/subfind_density.c/91
  27 2986 0             Right    126.8272   27328.4826  subfind_density()/subfind_density.c/92
 100 2976 0   DomainNodeIndex      0.3464   19507.9581  force_treeallocate()/forcetree.c/4800
 100 2977 0        Nodes_base   3737.5142   23245.4723  force_treeallocate()/forcetree.c/4803
  15 2989 0             DM_Vy    126.7521   27596.9150  subfind_density()/subfind_density.c/96
  15 2990 0             DM_Vz    126.7521   27723.6671  subfind_density()/subfind_density.c/97
  15 2991 0    	  DensDataGet    135.5194   27859.1865  subfind_density()/subfind_density.c/161
 112 2980 0            Father    221.9056   26805.6239  force_treeallocate()/forcetree.c/4827
 112 2981 0           Ngblist     63.4211   26869.0450  subfind_density()/subfind_density.c/78
 112 2982 0         Dist2list    126.8422   26995.8871  subfind_density()/subfind_density.c/79
  98 2980 0            Father    221.9056   26804.3073  force_treeallocate()/forcetree.c/4827
  98 2981 0           Ngblist     63.4208   26867.7280  subfind_density()/subfind_density.c/78
  98 2982 0         Dist2list    126.8416   26994.5696  subfind_density()/subfind_density.c/79
  27 2987 0              Todo     15.8534   27344.3360  subfind_density()/subfind_density.c/93
  27 2988 0             DM_Vx    126.8272   27471.1632  subfind_density()/subfind_density.c/95
  27 2989 0             DM_Vy    126.8272   27597.9904  subfind_density()/subfind_density.c/96
  27 2990 0             DM_Vz    126.8272   27724.8175  subfind_density()/subfind_density.c/97
 100 2978 0     Extnodes_base   3114.5952   26360.0675  force_treeallocate()/forcetree.c/4811
 100 2979 0          Nextnode    222.3014   26582.3690  force_treeallocate()/forcetree.c/4820
 100 2980 0            Father    221.9056   26804.2746  force_treeallocate()/forcetree.c/4827
  15 2992 0  	  DensDataResult    101.6396   27960.8261  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=15: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=39.1739 MB).
task 15: endrun called with an error level of 812


 112 2983 0    DataIndexTable     21.8182   27017.7053  subfind_density()/subfind_density.c/87
 112 2984 0      DataNodeList     58.1818   27075.8871  subfind_density()/subfind_density.c/89
 112 2985 0              Left    126.8422   27202.7293  subfind_density()/subfind_density.c/91
  98 2983 0    DataIndexTable     21.8182   27016.3878  subfind_density()/subfind_density.c/87
  98 2984 0      DataNodeList     58.1818   27074.5696  subfind_density()/subfind_density.c/89
  98 2985 0              Left    126.8416   27201.4111  subfind_density()/subfind_density.c/91
  27 2991 0    	  DensDataGet    117.8738   27842.6913  subfind_density()/subfind_density.c/161
  27 2992 0  	  DensDataResult     88.4053   27931.0966  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 100 2981 0           Ngblist     63.4190   26867.6935  subfind_density()/subfind_density.c/78
 100 2982 0         Dist2list    126.8380   26994.5315  subfind_density()/subfind_density.c/79
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 15
 112 2986 0             Right    126.8422   27329.5715  subfind_density()/subfind_density.c/92
 112 2987 0              Todo     15.8553   27345.4268  subfind_density()/subfind_density.c/93
 112 2988 0             DM_Vx    126.8422   27472.2689  subfind_density()/subfind_density.c/95
  98 2986 0             Right    126.8416   27328.2527  subfind_density()/subfind_density.c/92
  98 2987 0              Todo     15.8552   27344.1079  subfind_density()/subfind_density.c/93
  98 2988 0             DM_Vx    126.8416   27470.9494  subfind_density()/subfind_density.c/95

Task=27: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.9034 MB).
task 27: endrun called with an error level of 812


 100 2983 0    DataIndexTable     21.8182   27016.3497  subfind_density()/subfind_density.c/87
 100 2984 0      DataNodeList     58.1818   27074.5315  subfind_density()/subfind_density.c/89
 112 2989 0             DM_Vy    126.8422   27599.1111  subfind_density()/subfind_density.c/96
  98 2989 0             DM_Vy    126.8416   27597.7910  subfind_density()/subfind_density.c/96
  98 2990 0             DM_Vz    126.8416   27724.6325  subfind_density()/subfind_density.c/97
  98 2991 0    	  DensDataGet    125.7775   27850.4100  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 27
 100 2985 0              Left    126.8380   27201.3695  subfind_density()/subfind_density.c/91
 100 2986 0             Right    126.8380   27328.2075  subfind_density()/subfind_density.c/92
 112 2990 0             DM_Vz    126.8422   27725.9533  subfind_density()/subfind_density.c/97
  98 2992 0  	  DensDataResult     94.3331   27944.7432  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 100 2987 0              Todo     15.8548   27344.0622  subfind_density()/subfind_density.c/93
 112 2991 0    	  DensDataGet    143.3733   27869.3266  subfind_density()/subfind_density.c/161

Task=98: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.2568 MB).
task 98: endrun called with an error level of 812


 100 2988 0             DM_Vx    126.8380   27470.9002  subfind_density()/subfind_density.c/95
 112 2992 0  	  DensDataResult    107.5300   27976.8566  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 98
 100 2989 0             DM_Vy    126.8380   27597.7382  subfind_density()/subfind_density.c/96

Task=112: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=23.1434 MB).
task 112: endrun called with an error level of 812


 108 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 108 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 108 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 108 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 108 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 108 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 108 2973 0         FOF_PList    380.5269   18861.3322  fof_fof()/fof.c/249
 108 2974 0         FOF_GList    634.2115   19495.5437  fof_fof()/fof.c/337
 108 2975 0             Group     12.5684   19508.1121  fof_fof()/fof.c/380
 108 2976 0   DomainNodeIndex      0.3464   19508.4585  force_treeallocate()/forcetree.c/4800
 108 2977 0        Nodes_base   3737.5142   23245.9727  force_treeallocate()/forcetree.c/4803
 108 2978 0     Extnodes_base   3114.5952   26360.5679  force_treeallocate()/forcetree.c/4811
 100 2990 0             DM_Vz    126.8380   27724.5762  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 112
 108 2979 0          Nextnode    222.3014   26582.8693  force_treeallocate()/forcetree.c/4820
 108 2980 0            Father    221.9056   26804.7749  force_treeallocate()/forcetree.c/4827
 108 2981 0           Ngblist     63.4212   26868.1961  subfind_density()/subfind_density.c/78
 100 2991 0    	  DensDataGet    125.5095   27850.0857  subfind_density()/subfind_density.c/161
 An HDF5 error was detected, we'd better stop.
 108 2982 0         Dist2list    126.8423   26995.0384  subfind_density()/subfind_density.c/79
 108 2983 0    DataIndexTable     21.8182   27016.8565  subfind_density()/subfind_density.c/87
 108 2984 0      DataNodeList     58.1818   27075.0383  subfind_density()/subfind_density.c/89
 100 2992 0  	  DensDataResult     94.1321   27944.2178  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
HDF5-DIAG: Error detected in HDF5 (1.8.12)  108 2985 0              Left    126.8423   27201.8806  subfind_density()/subfind_density.c/91
 108 2986 0             Right    126.8423   27328.7229  subfind_density()/subfind_density.c/92
 108 2987 0              Todo     15.8553   27344.5782  subfind_density()/subfind_density.c/93

Task=100: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.7822 MB).
task 100: endrun called with an error level of 812


MPI-process 112:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
 108 2988 0             DM_Vx    126.8423   27471.4205  subfind_density()/subfind_density.c/95
 108 2989 0             DM_Vy    126.8423   27598.2628  subfind_density()/subfind_density.c/96
 108 2990 0             DM_Vz    126.8423   27725.1051  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 100
task 112: endrun called with an error level of 124


 108 2991 0    	  DensDataGet    119.6047   27844.7098  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 112
 108 2992 0  	  DensDataResult     89.7035   27934.4133  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=108: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.5867 MB).
task 108: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 108
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  99 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  99 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  99 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  99 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  99 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  99 2973 0         FOF_PList    380.4394   18861.2447  fof_fof()/fof.c/249
  99 2974 0         FOF_GList    634.0656   19495.3104  fof_fof()/fof.c/337
  99 2975 0             Group     12.9907   19508.3011  fof_fof()/fof.c/380
  99 2976 0   DomainNodeIndex      0.3464   19508.6475  force_treeallocate()/forcetree.c/4800
  99 2977 0        Nodes_base   3737.5142   23246.1617  force_treeallocate()/forcetree.c/4803
  99 2978 0     Extnodes_base   3114.5952   26360.7569  force_treeallocate()/forcetree.c/4811
  99 2979 0          Nextnode    222.3014   26583.0583  force_treeallocate()/forcetree.c/4820
  99 2980 0            Father    221.9056   26804.9639  force_treeallocate()/forcetree.c/4827
  99 2981 0           Ngblist     63.4066   26868.3704  subfind_density()/subfind_density.c/78
  99 2982 0         Dist2list    126.8131   26995.1836  subfind_density()/subfind_density.c/79
  99 2983 0    DataIndexTable     21.8182   27017.0017  subfind_density()/subfind_density.c/87
  99 2984 0      DataNodeList     58.1818   27075.1835  subfind_density()/subfind_density.c/89
  99 2985 0              Left    126.8131   27201.9967  subfind_density()/subfind_density.c/91
  99 2986 0             Right    126.8131   27328.8098  subfind_density()/subfind_density.c/92
  99 2987 0              Todo     15.8516   27344.6614  subfind_density()/subfind_density.c/93
  99 2988 0             DM_Vx    126.8131   27471.4746  subfind_density()/subfind_density.c/95
  99 2989 0             DM_Vy    126.8131   27598.2877  subfind_density()/subfind_density.c/96
  99 2990 0             DM_Vz    126.8131   27725.1008  subfind_density()/subfind_density.c/97
  99 2991 0    	  DensDataGet    116.9062   27842.0070  subfind_density()/subfind_density.c/161
  99 2992 0  	  DensDataResult     87.6796   27929.6866  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=99: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.3134 MB).
task 99: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 99
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  30 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  30 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  30 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  30 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  30 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  30 2973 0         FOF_PList    380.5332   18861.3386  fof_fof()/fof.c/249
  30 2974 0         FOF_GList    634.2221   19495.5607  fof_fof()/fof.c/337
  30 2975 0             Group     12.2250   19507.7856  fof_fof()/fof.c/380
  30 2976 0   DomainNodeIndex      0.3464   19508.1320  force_treeallocate()/forcetree.c/4800
  30 2977 0        Nodes_base   3737.5142   23245.6462  force_treeallocate()/forcetree.c/4803
  30 2978 0     Extnodes_base   3114.5952   26360.2414  force_treeallocate()/forcetree.c/4811
  30 2979 0          Nextnode    222.3014   26582.5429  force_treeallocate()/forcetree.c/4820
  30 2980 0            Father    221.9056   26804.4484  force_treeallocate()/forcetree.c/4827
  30 2981 0           Ngblist     63.4222   26867.8707  subfind_density()/subfind_density.c/78
  30 2982 0         Dist2list    126.8444   26994.7151  subfind_density()/subfind_density.c/79
  30 2983 0    DataIndexTable     21.8182   27016.5332  subfind_density()/subfind_density.c/87
  30 2984 0      DataNodeList     58.1818   27074.7150  subfind_density()/subfind_density.c/89
  30 2985 0              Left    126.8444   27201.5594  subfind_density()/subfind_density.c/91
  30 2986 0             Right    126.8444   27328.4039  subfind_density()/subfind_density.c/92
  30 2987 0              Todo     15.8556   27344.2594  subfind_density()/subfind_density.c/93
  30 2988 0             DM_Vx    126.8444   27471.1038  subfind_density()/subfind_density.c/95
  30 2989 0             DM_Vy    126.8444   27597.9482  subfind_density()/subfind_density.c/96
  30 2990 0             DM_Vz    126.8444   27724.7927  subfind_density()/subfind_density.c/97
  30 2991 0    	  DensDataGet    144.1486   27868.9412  subfind_density()/subfind_density.c/161
  30 2992 0  	  DensDataResult    108.1114   27977.0526  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=30: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=22.9474 MB).
task 30: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 30
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 109 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 109 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 109 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 109 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 109 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 109 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
 109 2973 0         FOF_PList    380.2300   18861.0354  fof_fof()/fof.c/249
 109 2974 0         FOF_GList    633.7167   19494.7521  fof_fof()/fof.c/337
 109 2975 0             Group     13.3798   19508.1320  fof_fof()/fof.c/380
 109 2976 0   DomainNodeIndex      0.3464   19508.4783  force_treeallocate()/forcetree.c/4800
 109 2977 0        Nodes_base   3737.5142   23245.9925  force_treeallocate()/forcetree.c/4803
 109 2978 0     Extnodes_base   3114.5952   26360.5877  force_treeallocate()/forcetree.c/4811
 109 2979 0          Nextnode    222.3014   26582.8892  force_treeallocate()/forcetree.c/4820
 109 2980 0            Father    221.9056   26804.7948  force_treeallocate()/forcetree.c/4827
 109 2981 0           Ngblist     63.3717   26868.1664  subfind_density()/subfind_density.c/78
 109 2982 0         Dist2list    126.7433   26994.9098  subfind_density()/subfind_density.c/79
 109 2983 0    DataIndexTable     21.8182   27016.7280  subfind_density()/subfind_density.c/87
 109 2984 0      DataNodeList     58.1818   27074.9097  subfind_density()/subfind_density.c/89
 109 2985 0              Left    126.7433   27201.6531  subfind_density()/subfind_density.c/91
 109 2986 0             Right    126.7433   27328.3964  subfind_density()/subfind_density.c/92
 109 2987 0              Todo     15.8429   27344.2394  subfind_density()/subfind_density.c/93
 109 2988 0             DM_Vx    126.7433   27470.9827  subfind_density()/subfind_density.c/95
 109 2989 0             DM_Vy    126.7433   27597.7261  subfind_density()/subfind_density.c/96
 109 2990 0             DM_Vz    126.7433   27724.4694  subfind_density()/subfind_density.c/97
 109 2991 0    	  DensDataGet    114.3734   27838.8428  subfind_density()/subfind_density.c/161
 109 2992 0  	  DensDataResult     85.7800   27924.6228  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=109: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.3772 MB).
task 109: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 109
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  96 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  96 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  96 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  96 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  96 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  96 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  96 2973 0         FOF_PList    380.4548   18861.2601  fof_fof()/fof.c/249
  96 2974 0         FOF_GList    634.0913   19495.3514  fof_fof()/fof.c/337
  31 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  31 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  31 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  31 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  31 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  31 2972 0          TopNodes      5.1460   18480.8054  domain_allocate()/domain.c/570
  31 2973 0         FOF_PList    380.4080   18861.2133  fof_fof()/fof.c/249
  31 2974 0         FOF_GList    634.0133   19495.2266  fof_fof()/fof.c/337
  31 2975 0             Group     10.7339   19505.9605  fof_fof()/fof.c/380
  31 2976 0   DomainNodeIndex      0.3464   19506.3069  force_treeallocate()/forcetree.c/4800
  31 2977 0        Nodes_base   3737.5142   23243.8211  force_treeallocate()/forcetree.c/4803
  31 2978 0     Extnodes_base   3114.5952   26358.4163  force_treeallocate()/forcetree.c/4811
  31 2979 0          Nextnode    222.3014   26580.7177  force_treeallocate()/forcetree.c/4820
  31 2980 0            Father    221.9056   26802.6233  force_treeallocate()/forcetree.c/4827
  31 2981 0           Ngblist     63.4013   26866.0246  subfind_density()/subfind_density.c/78
  96 2975 0             Group     12.1990   19507.5504  fof_fof()/fof.c/380
  96 2976 0   DomainNodeIndex      0.3464   19507.8968  force_treeallocate()/forcetree.c/4800
  31 2982 0         Dist2list    126.8027   26992.8273  subfind_density()/subfind_density.c/79
  31 2983 0    DataIndexTable     21.8182   27014.6455  subfind_density()/subfind_density.c/87
  31 2984 0      DataNodeList     58.1818   27072.8273  subfind_density()/subfind_density.c/89
  31 2985 0              Left    126.8027   27199.6299  subfind_density()/subfind_density.c/91
  96 2977 0        Nodes_base   3737.5142   23245.4110  force_treeallocate()/forcetree.c/4803
  96 2978 0     Extnodes_base   3114.5952   26360.0062  force_treeallocate()/forcetree.c/4811
  96 2979 0          Nextnode    222.3014   26582.3076  force_treeallocate()/forcetree.c/4820
  31 2986 0             Right    126.8027   27326.4326  subfind_density()/subfind_density.c/92
  31 2987 0              Todo     15.8503   27342.2829  subfind_density()/subfind_density.c/93
  31 2988 0             DM_Vx    126.8027   27469.0855  subfind_density()/subfind_density.c/95
  96 2980 0            Father    221.9056   26804.2132  force_treeallocate()/forcetree.c/4827
  96 2981 0           Ngblist     63.4091   26867.6223  subfind_density()/subfind_density.c/78
  96 2982 0         Dist2list    126.8183   26994.4406  subfind_density()/subfind_density.c/79
  31 2989 0             DM_Vy    126.8027   27595.8882  subfind_density()/subfind_density.c/96
  31 2990 0             DM_Vz    126.8027   27722.6908  subfind_density()/subfind_density.c/97
  31 2991 0    	  DensDataGet    127.1541   27849.8450  subfind_density()/subfind_density.c/161
  96 2983 0    DataIndexTable     21.8182   27016.2588  subfind_density()/subfind_density.c/87
  96 2984 0      DataNodeList     58.1818   27074.4406  subfind_density()/subfind_density.c/89
  31 2992 0  	  DensDataResult     95.3656   27945.2105  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=31: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.7895 MB).
task 31: endrun called with an error level of 812


  96 2985 0              Left    126.8183   27201.2588  subfind_density()/subfind_density.c/91
  96 2986 0             Right    126.8183   27328.0771  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 31
  96 2987 0              Todo     15.8523   27343.9294  subfind_density()/subfind_density.c/93
  96 2988 0             DM_Vx    126.8183   27470.7476  subfind_density()/subfind_density.c/95
  96 2989 0             DM_Vy    126.8183   27597.5659  subfind_density()/subfind_density.c/96
  96 2990 0             DM_Vz    126.8183   27724.3841  subfind_density()/subfind_density.c/97
  96 2991 0    	  DensDataGet    117.3047   27841.6889  subfind_density()/subfind_density.c/161
  96 2992 0  	  DensDataResult     87.9786   27929.6674  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=96: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.3326 MB).
task 96: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 96
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 96:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
task 96: endrun called with an error level of 124


    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 96
-------------------------------------------------------------------------------------------
AvailMem:	 Largest =   63110.91 Mb (on task=120), Smallest =   62841.85 Mb (on task=0), Average =   63040.03 Mb
Total Mem:	 Largest =   64532.40 Mb (on task=24), Smallest =   64531.77 Mb (on task=0), Average =   64532.08 Mb
Committed_AS:	 Largest =    1689.92 Mb (on task=0), Smallest =    1421.15 Mb (on task=120), Average =    1492.05 Mb
SwapTotal:	 Largest =   16384.00 Mb (on task=0), Smallest =   16384.00 Mb (on task=0), Average =   16384.00 Mb
SwapFree:	 Largest =   16384.00 Mb (on task=96), Smallest =   16262.84 Mb (on task=14), Average =   16345.11 Mb
AllocMem:	 Largest =    1689.92 Mb (on task=0), Smallest =    1421.15 Mb (on task=120), Average =    1492.05 Mb
-------------------------------------------------------------------------------------------
Task with the maximum commited memoryodin077

This is P-Gadget, version 3.0.

Running on 128 MPI tasks.

Code was compiled with settings:

        PERIODIC
        UNEQUALSOFTENINGS
        PMGRID=1024
        ASMTH=1.25
        RCUT=4.5
        PLACEHIGHRESREGION=1+2+16
        ENLARGEREGION=1.2
        MULTIPLEDOMAINS=64
        PEANOHILBERT
        WALLCLOCK
        MYSORT
        DOUBLEPRECISION
        SUBFIND
        DENSITY_SPLIT_BY_TYPE=1+2+16+32
        NO_ISEND_IRECV_IN_DOMAIN
        FIX_PATHSCALE_MPI_STATUS_IGNORE_BUG
        NOSTOP_WHEN_BELOW_MINTIMESTEP
        OVERRIDE_STOP_FOR_SUBOPTIMUM_DOMAINS
        LONGIDS
        GENERATE_GAS_IN_ICS
        HAVE_HDF5
        EAGLE
        EAGLE_COOLING
        EAGLE_SFR
        EAGLE_STELLAR_FEEDBACK
        EAGLE_STELLAR_FEEDBACK_METALS
        EAGLE_TIMESTEP
        EAGLE_TIMESTEP_LIMITER
        EAGLE_BH_TIMESTEP
        EAGLE_SFR_METALLICITY_DEPENDENT_SF_THRESHOLD
        EAGLE_METALSMOOTHING
        EAGLE_SNIA_IRON
        EAGLE_TRACK_SNIA_MASS
        EAGLE_TRACK_SNII_MASS
        EAGLE_TRACK_AGB_MASS
        EAGLE_SAMPLE_STELLAREVOLUTION
        EAGLE_BH
        BLACK_HOLES
        DETACH_BLACK_HOLES
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        EAGLE_BH_ACCRETION
        ENFORCE_EDDINGTON_LIMIT
        BONDI
        SWALLOWGAS
        EVALPOTENTIAL
        REPOSITION_ON_POTMIN
        BH_USE_GASVEL_IN_BONDI
        EAGLE_BH_THERMALFEEDBACK
        EAGLE_BH_OWLS_MERGERS
        EAGLE_BH_MAX_MERGING_DISTANCE
        EAGLE_BH_MAX_REPOSITION_DISTANCE
        BH_ANG_MOM_ACCRETION
        RGB_EAGLE_FEEDBACK
        EAGLE_Z_DEP_FEEDBACK
        EAGLE_DETACH_BH_DENSITY
        EAGLE_PE_AGN_FIX_MS
        FOF
        FOF_PRIMARY_LINK_TYPES=2
        FOF_SECONDARY_LINK_TYPES=1+16+32
        EAGLE_ASSIGN_HOSTHALO_MASS
        EAGLE_IO
        EAGLE_OWLS_SNAPSHOT
        EAGLE_HDF5_COMPRESSED_SNAPSHOT
        EAGLE_EXTRA_ARRAYS
        EAGLE_SNIPSHOTS
        EAGLE_SORT_OUTPUT
        EAGLE_SUPPRESS_MEMORY_TABLE_DUMP
        EAGLE_OUTPUT_GRID
        EAGLE_OUTPUT_LOS
        MyMaxLosBuffer=4096
        OUTPUTLINEOFSIGHT_PARTICLES
        ANARCHY_SPH
        ANARCHY_SPH_PRESSURE_ENTROPY
        SPH_KERNEL_C2
        HYDRANGEA_REDUCED_SNIPSHOT_CONTENT
        HYDRANGEA_SUPPRESS_HASH_IN_SNIPSHOTS
svn version:exported

Size of particle structure       320  [bytes]

Size of sph particle structure   504  [bytes]

Size of star particle structure 384  [bytes]


Size of BH particle structure    256  [bytes]

Obtaining parameters from file 'eagle.param.subfind.ODIN-N2':
RunLabel                           "HaloF22_DM-N1024_ID67"
InitCondFile                       ../ICs/snap_000
OutputDir                          data
EnergyFile                         energy.txt
InfoFile                           info.txt
TimingsFile                        timings.txt
CpuFile                            cpu.txt
TimebinFile                        timebins.txt
SnapshotFileBase                   snap
SmallSnapshotFileBase              snip
RestartFile                        restart
CoolTablePath                      /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/
YieldTablePath                     /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/YieldTables/
TimeLimitCPU                       96000
ResubmitOn                         0
ResubmitCommand                    ./autoresubmit.sh
MaxMemSize                         28000
SnapFormat                         3
FoFFormat                          3
ICFormat                           3
NumFilesPerSnapshot                16
NumFilesWrittenInParallel          16
SmallOutputTar                     1
ComovingIntegrationOn              1
CoolingOn                          1
MetDepCoolingOn                    1
StarformationOn                    1
StellarEnergyFeedbackOn            1
StellarMetalFeedbackOn             1
SNII_MassTransferOn                1
SNII_EnergyTransferOn              1
SNII_WindIsotropicOn               1
SNIa_MassTransferOn                1
SNIa_EnergyTransferOn              1
SNIa_EnergyTransferStochastic      1
AGB_MassTransferOn                 1
AGB_EnergyTransferOn               1
StellarEvol_FeedbackOn             1
PartAllocFactor                    3.5
MetAllocFactor                     0.25
BHAllocFactor                      0.05
BufferSize                         400
TimeBegin                          0.0078125
TimeMax                            1
UnitLength_in_cm                   3.08568e+24
UnitMass_in_g                      1.989e+43
UnitVelocity_in_cm_per_s           100000
GravityConstantInternal            0
PeriodicBoundariesOn               1
BoxSize                            2168.64
Omega0                             0.307
OmegaLambda                        0.693
OmegaBaryon                        0.0482519
HubbleParam                        0.6777
MinGasHsmlFractional               0.1
SofteningGas                       0.00180239
SofteningHalo                      0.00180239
SofteningDisk                      0.01
SofteningBulge                     0.05
SofteningStars                     0.00180239
SofteningBndry                     0.00180239
SofteningGasMaxPhys                0.00047439
SofteningHaloMaxPhys               0.00047439
SofteningDiskMaxPhys               0.0025
SofteningBulgeMaxPhys              0.0125
SofteningStarsMaxPhys              0.00047439
SofteningBndryMaxPhys              0.00047439
OutputListOn                       1
OutputListFilename                 /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snapshots_plus.dat
TimeBetSnapshot                    0
TimeOfFirstSnapshot                0
SmallOutputListOn                  1
ExtraOutputListFilename            /ptmp/mpa/ybahe/HYDRANGEA/OutputLists/hydrangea_snipshots_default_long.dat
TimeBetSmallSnapshot               0
TimeOfFirstSmallSnapshot           0
FoFOutputListOn                    1
TimeBetOnTheFlyFoF                 1.005
TimeOfFirstGridOutput              0.05
TimeBetGridOutput                  1.025
TimeBetLineOfSight                 1.01
TimeOfFirstLineOfSight             0.09
CpuTimeBetRestartFile              22000
TimeBetStatistics                  0.02
MaxRMSDisplacementFac              0.25
ErrTolThetaSubfind                 0.6
DesLinkNgb                         20
TypeOfTimestepCriterion            0
ErrTolIntAccuracy                  0.025
MaxSizeTimestep                    0.01
MinSizeTimestep                    6e-08
ErrTolTheta                        0.6
TypeOfOpeningCriterion             1
ErrTolForceAcc                     0.005
TreeDomainUpdateFrequency          0.005
DesNumNgb                          58
DesNumNgbStar                      48
DesNumNgbYoungStar                 48
MaxNumNgbDeviation                 1
CourantFac                         0.15
MaxSmoothingLengthChange           1.26
ArtBulkViscConst                   2
ArtBulkViscConstMin                0.05
ArtDiffConst                       1
ArtDiffConstMin                    0
MinGasTemp                         100
InitGasTemp                        268.7
InitMetallicity                    0
InitAbundance_Hydrogen             0.752
InitAbundance_Helium               0.248
InitAbundance_Carbon               0
InitAbundance_Nitrogen             0
InitAbundance_Oxygen               0
InitAbundance_Neon                 0
InitAbundance_Magnesium            0
InitAbundance_Silicon              0
InitAbundance_Iron                 0
CalciumOverSilicon                 0.0941736
SulphurOverSilicon                 0.605416
REION_H_ZCenter                    11.5
REION_H_Heating_EVpH               2
REION_He_ZCenter                   3.5
REION_He_ZSigma                    0.5
REION_He_Heating_EVpH              2
Generations                        1
SF_SchmidtLawCoeff_MSUNpYRpKPC2    0.0001515
SF_SchmidtLawExponent              1.4
SF_SchmidtLawHighDensExponent      2
SF_SchmidtLawHighDensThresh_HpCM3  1000
SF_THRESH_MaxPhysDensOn            0
SF_THRESH_MaxPhysDens_HpCM3        100000
SF_THRESH_MinOverDens              57.7
SF_THRESH_MinPhysDens_HpCM3        0.1
SF_THRESH_TempMargin_DEX           0.5
SF_THRESH_MetDepSFThreshNorm_HpCM3 0.1
SF_THRESH_MetDepSFThreshSlope      -0.64
SF_THRESH_MetDepSFThreshMaxThresh_HpCM310
EOS_NormPhysDens_HpCM3             0.1
EOS_Jeans_MinPhysDens_HpCM3        0.1
EOS_Jeans_MinOverDens              10
EOS_Jeans_GammaEffective           1.33333
EOS_Jeans_TempNorm_K               8000
EOS_Cool_MinOverDens               10
EOS_Cool_MinPhysDens_HpCM3         1e-05
EOS_Cool_GammaEffective            1
EOS_Cool_TempNorm_K                8000
IMF_Model                          Chabrier
IMF_Exponent                       0
IMF_LifetimeModel                  P98
IMF_MinMass_MSUN                   0.1
IMF_MaxMass_MSUN                   100
StellarEvolutionCut_Gyr            0.1
StellarEvolutionTimestepInterval   10
StellarEvolutionCut_Gyr_2          1
StellarEvolutionTimestepInterval_2 100
SNIa_Model                         Efolding
SNIa_Energy_ERG                    1e+51
SNIa_EnergyFraction                1
SNIa_Efficiency                    0.002
SNIa_TimeScale                     2
SNIa_EjectaVelocity_KMpS           10
SNII_MinMass_MSUN                  6
SNII_MaxMass_MSUN                  100
SNII_Factor_Hydrogen               1
SNII_Factor_Helium                 1
SNII_Factor_Carbon                 0.5
SNII_Factor_Nitrogen               1
SNII_Factor_Oxygen                 1
SNII_Factor_Neon                   1
SNII_Factor_Magnesium              2
SNII_Factor_Silicon                1
SNII_Factor_Iron                   0.5
SNII_MinEnergyFraction             0.3
SNII_MaxEnergyFraction             3
SNII_Tvir0_K                       0.00126637
SNII_Width_logTvir_dex             1
SNII_Energy_ERG                    1e+51
SNII_WindDelay_YR                  3e+07
stellar_feedback_mode              thermal
stellar_feedback_DeltaT            constant
stellar_feedback_tvir              vel_disp
SNII_Delta_T_K                     3.16228e+07
SNII_Delta_T_Divided_By_T_Vir      0
SNII_Min_Delta_T_K                 0
SNII_Max_Delta_T_K                 0
SNII_normalisation_Delta_T_K       0
SNII_exponent_Delta_T              0
massDMpart                         -1
BlackHoleNgbFactor                 1
BlackHoleNumberOfNeighboursToHeat  1
BlackHoleAccretionFactor           1
BlackHoleEddingtonFactor           1
BlackHoleAccretionSlope            0
BlackHoleRadiativeEfficiency       0.1
BlackHoleFeedbackFactor            0.15
SeedBlackHoleMass_Msun             100000
MinFoFMassForNewSeed_Msun          1e+10
BlackHoleMaxAccretionRadius        1e+10
BH_MaxRepositionDistanceFactor     3
BH_MaxMergingDistanceFactor        3
BH_maxHeatingProbability           0.3
BH_feedback_mode                   constant
BH_ConstantHeatTemp                1e+09
BH_MinHeatTemp                     0
BH_MaxHeatTemp                     0
BH_MinHeatLimit                    0
BH_MaxHeatLimit                    0
BlackHoleViscousAlpha              10000
SNII_zdep_power                    0
SNII_rhogas_power                  1
SNII_rhogas_physdensnormfac        6.7


found 30 times in output-list.

found 178 times in small output-list.

found 178 times in fof output-list.

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43 
UnitTime_in_s = 3.08568e+19 
UnitVelocity_in_cm_per_s = 100000 
UnitDensity_in_cgs = 6.76991e-31 
UnitEnergy_in_cgs = 1.989e+53 

initialize Ewald correction...

reading Ewald tables from file `ewald_spc_table_64_dbl.dat'
initialization of periodic boundaries finished.
EOS normalisation temperature is  8.000000e+03 K, converted to internal energy 8.063068e+11
Cooling EOS normalisation temp is  8.000000e+03 K, converted to internal energy 8.063068e+11
The Jeans mass (assuming a neutral, primordial gas) at the EOS threshold is 9.156013e+06 Msun
Chemistry initialized.
 Stellar Feedback mode is THERMAL
 Stellar Feedback virial temperature calculation is from local dm velocity dispersion. 
 Stellar Feedback heating temperature is constant, and equal to 3.162280e+07 [K]
SNIa model: Efolding, set model to 2
Setting lifetime model to P98
IMF normalization norm=9.999885e-01
IMF initialized.
Computing yield for Hydrogen	 index=0
Computing yield for Helium	 index=1
Computing yield for Carbon	 index=2
Computing yield for Nitrogen	 index=3
Computing yield for Oxygen	 index=4
Computing yield for Neon	 index=5
Computing yield for Magnesium	 index=6
Computing yield for Silicon	 index=7
Computing yield for Iron	 index=8
Computing ejecta
Yields initialized.
 feedback parameters: for your IMF, the number of SNe if type II per solar mass formed = 0.017362, the energy of a single SNe = 1.000000e+51 [erg]
 Using 100 percent of SNII energy, DeltaT = 1.000000e+06 corresponds to a wind speed of 204.574409 km/s and mass loading of 41.715274
 Using 100 percent of SNII energy, DeltaT = 1.000000e+07 corresponds to a wind speed of 646.921083 km/s and mass loading of 4.171527
 using 100 percent of SNII energy, DeltaT = 1.000000e+08 corresponds to a wind speed of 2045.744088 km/s and mass loading of 0.417153
 Using 100 percent of SNII energy, DeltaT = 3.162280e+07 corresponds to a wind speed of 1150.406867 km/s and mass loading of 1.319152
 Black hole feedback temperature will be CONSTANT
Done with cooling table header.


Solar abundances:

Solar abundance of Hydrogen = 0.706498
Solar abundance of Helium = 0.280555
Solar abundance of Carbon = 0.00206654
Solar abundance of Nitrogen = 0.000835626
Solar abundance of Oxygen = 0.00549262
Solar abundance of Neon = 0.00141446
Solar abundance of Magnesium = 0.000590706
Solar abundance of Silicon = 0.000682587
Solar abundance of Sulphur = 0.000408985
Solar abundance of Calcium = 6.4355e-05
Solar abundance of Iron = 0.00110322

SolarMetallicity = 0.0129468
Hydrogen reionisation heating = 1.915792e+12 erg/g
  Helium reionisation heating = 1.915792e+12 erg/g

Allocated 18473.6 MByte for particle storage.


reading file `data//snapshot_021_z000p297/snap_021_z000p297.0.hdf5' on task=0 (contains 134745190 particles.)
distributing this file to tasks 0-7
Type 0 (gas):          0  (tot=     0000000000) masstab=0
Type 1 (halo):  134697702  (tot=     2124916242) masstab=0.000779539
Type 2 (disk):      8467  (tot=     0000134208) masstab=0
Type 3 (bulge):    39021  (tot=     0002354161) masstab=0
Type 4 (stars):        0  (tot=     0000000000) masstab=0
Type 5 (bndry):        0  (tot=     0000000000) masstab=0

 Converting element abundances to metal masses ... 
reading done.
Total number of particles :  2127404611


Setting next time for grid output file to Time_next= 0.775031

Setting next time for line of sight file to Time_next= 0.772075

Setting next time for snipshot file to Time_next= 0.77857

Setting next time for fof output to Time_next= 0.77857
 init.c: need to read some more variables from snapshot -- currently zeroed
Testing ID uniqueness...
success.  took=11.7265 sec
Warning: You have compiled with EAGLE, but are running a DM only simulation. Hope you intended this .. 
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)

At 'DOMAIN', domain_Decomposition()/domain.c/310: Largest Allocation = 18978.1 Mbyte (on task=1), Smallest = 18978.1 Mbyte, Average = 18978.1 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes     23.0782   18498.7375  domain_allocate()/domain.c/570
   1 2973 0        domain_key    443.8112   18942.5487  domain_Decomposition()/domain.c/231
   1 2984 0        domainWork      1.7752   18944.3298  domain_Decomposition()/domain.c/254
   1 2985 0     domainWorkSph      1.7752   18946.1051  domain_Decomposition()/domain.c/256
   1 2986 0       domainCount      1.7752   18947.8803  domain_Decomposition()/domain.c/258
   1 2987 0    domainCountSph      1.7752   18949.6555  domain_Decomposition()/domain.c/260
   1 2992 0  domainCountStars      1.7752   18951.4327  domain_Decomposition()/domain.c/272
   1 2997 0    domainCountBHs      1.7752   18953.2099  domain_Decomposition()/domain.c/285
   1 2998 0          topNodes     24.8534   18978.0634  domain_Decomposition()/domain.c/290
----------------------------------------------------------------------------------------
Before=103777
After=104313
NTopleaves= 91274  NTopnodes=104313 (space for 465370)
gravity work-load balance=1.00036   memory-balance=1.00036   SPH work-load balance=0
iter=0 exchange of 2111364579 particles (ret=0)
domain decomposition done. (took 29.151 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5279 MByte in top-level domain structure

Allocated 7296.41 MByte for BH-tree, and 0.348183 Mbyte for top-leaves.  (presently allocted 25777.6 MB)

Begin Ngb-tree construction.
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

Computing the weighted densities to get the temperatures...
Ngb-Tree contruction finished 
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...
Computing the predicted entropy from the internal energies read in the file...
Computing the weighted densities to get the temperatures...

At 'SPH_DENSITY', density()/density.c/935: Largest Allocation = 26120.4 Mbyte (on task=59), Smallest = 26119.8 Mbyte, Average = 26120.3 Mbyte

------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  59 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  59 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  59 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  59 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  59 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  59 2972 0          TopNodes      5.1730   18480.8323  domain_allocate()/domain.c/570
  59 2973 0   DomainNodeIndex      0.3482   18481.1805  force_treeallocate()/forcetree.c/4800
  59 2974 0        Nodes_base   3737.5640   22218.7445  force_treeallocate()/forcetree.c/4803
  59 2975 0     Extnodes_base   3114.6367   25333.3812  force_treeallocate()/forcetree.c/4811
  59 2976 0          Nextnode    222.3035   25555.6847  force_treeallocate()/forcetree.c/4820
  59 2977 0            Father    221.9056   25777.5903  force_treeallocate()/forcetree.c/4827
  59 2978 0           Ngblist     63.4243   25841.0147  density()/density.c/690
  59 2979 0              Left    126.8487   25967.8633  density()/density.c/692
  59 2980 0             Right    126.8487   26094.7120  density()/density.c/693
  59 2981 0    DataIndexTable      7.0175   26101.7296  density()/density.c/738
  59 2982 0      DataNodeList     18.7134   26120.4430  density()/density.c/740
----------------------------------------------------------------------------------------
GetCoolingTables: Redshift 1 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.271.hdf5
GetCoolingTables: Redshift 2 /ptmp/mpa/ybahe/HYDRANGEA/BG_Tables/CoolingTables/z_0.333.hdf5

Begin to compute FoF group catalogues...  (presently allocated=25777.6 MB)
Blackholes: Eliminated 0 gas particles and merged away 0 black holes.
domain decomposition... LevelToTimeBin[TakeLevel=0]=0  (presently allocated=18475.6 MB)
Allocated 23.1407 MByte for top-level domain structure
use of 479.326 MB of temporary storage for domain decomposition... (presently allocated=18978.1 MB)
Before=94777
After=104801
NTopleaves= 91701  NTopnodes=104801 (space for 465370)
gravity work-load balance=1.00064   memory-balance=1.00064   SPH work-load balance=0
iter=0 exchange of 2113073201 particles (ret=0)
domain decomposition done. (took 33.4207 sec)
begin Peano-Hilbert order...
Peano-Hilbert done.
Freed 16.5055 MByte in top-level domain structure

Comoving linking length: 0.00901189    (presently allocated=18480.9 MB)
Tree construction.

Start linking particles (presently allocated=26538.7 MB)
links on local processor done (took 148.041 sec).
Marked=0207730353 out of the 2124916242 primaries which are linked

linking across processors (presently allocated=26862.8 MB) 
have done 0002847660 cross links (processed 0207730353, took 19.1057 sec)
have done 0001250309 cross links (processed 0170035314, took 17.8385 sec)
have done 0000888824 cross links (processed 0143440040, took 17.3003 sec)
have done 0000558949 cross links (processed 0097903228, took 14.8679 sec)
have done 0000351019 cross links (processed 0069735129, took 14.2179 sec)
have done 0000216289 cross links (processed 0040341614, took 13.6794 sec)
have done 0000126291 cross links (processed 0016752876, took 6.5244 sec)
have done 0000072910 cross links (processed 0005952523, took 2.12644 sec)
have done 0000044121 cross links (processed 0004083477, took 2.07327 sec)
have done 0000025699 cross links (processed 0002004471, took 1.34628 sec)
have done 0000013462 cross links (processed 0001339393, took 1.92483 sec)
have done 0000007984 cross links (processed 0001388668, took 1.94253 sec)
have done 0000004250 cross links (processed 0000490994, took 1.40735 sec)
have done 0000001412 cross links (processed 0000054029, took 0.395725 sec)
have done 0000000463 cross links (processed 0000007189, took 0.365084 sec)
have done 0000000220 cross links (processed 0000002203, took 0.360335 sec)
have done 0000000081 cross links (processed 0000001300, took 0.360732 sec)
have done 0000000027 cross links (processed 0000000541, took 0.360366 sec)
have done 0000000006 cross links (processed 0000000111, took 0.359855 sec)
have done 0000000000 cross links (processed 0000000014, took 0.360606 sec)
Local groups found.

group finding took = 265.458 sec
Start finding nearest dm-particle (presently allocated=26538.7 MB)
still finding nearest... (presently allocated=26806.2 MB)
done finding nearest dm-particle
attaching gas and star particles to nearest dm particles took = 0.80924 sec
compiling local group data and catalogue took = 9.37304 sec

Total number of groups with at least 32 particles: 1347009
Largest group has 123617800 particles.
Total number of particles in groups: 1246058088

group properties are now allocated.. (presently allocated=19507.4 MB)
computation of group properties took = 0.512078 sec
start global sorting of group catalogues number 21
LargestGroup = 96364.895879; SmallestGroup = 0.024945
tagging gas particles with halo masses took = 0.218809 sec
Group catalogues globally sorted. took = 9.87325 sec
starting saving of group catalogue number 21
data//groups_021_z000p297
Group catalogues number 21 saved. took = 17.0412 sec

We now execute a parallel version of SUBFIND.
Tree construction.
tree build took 12.3635 sec
finding densities, mode 0 
final operations ... 
density and smoothing length for species 0 took 2.06828 sec
Tree construction for species 1 (2124916242).
tree build for species 1 took 12.3452 sec
finding densities, mode 1 
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   9 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   9 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   9 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   9 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   9 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   9 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   9 2973 0         FOF_PList    380.5849   18861.4414  fof_fof()/fof.c/249
   9 2974 0         FOF_GList    634.3081   19495.7495  fof_fof()/fof.c/337
   9 2975 0             Group     12.5311   19508.2806  fof_fof()/fof.c/380
   9 2976 0   DomainNodeIndex      0.3498   19508.6304  force_treeallocate()/forcetree.c/4800
   9 2977 0        Nodes_base   3737.6087   23246.2391  force_treeallocate()/forcetree.c/4803
   9 2978 0     Extnodes_base   3114.6739   26360.9130  force_treeallocate()/forcetree.c/4811
   9 2979 0          Nextnode    222.3054   26583.2184  force_treeallocate()/forcetree.c/4820
   9 2980 0            Father    221.9056   26805.1240  force_treeallocate()/forcetree.c/4827
   9 2981 0           Ngblist     63.4308   26868.5548  subfind_density()/subfind_density.c/78
   9 2982 0         Dist2list    126.8616   26995.4165  subfind_density()/subfind_density.c/79
   9 2983 0    DataIndexTable     21.8182   27017.2346  subfind_density()/subfind_density.c/87
 118 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 118 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 118 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 118 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 118 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 118 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 118 2973 0         FOF_PList    380.4070   18861.2635  fof_fof()/fof.c/249
 118 2974 0         FOF_GList    634.0117   19495.2752  fof_fof()/fof.c/337
 118 2975 0             Group     10.9197   19506.1949  fof_fof()/fof.c/380
 118 2976 0   DomainNodeIndex      0.3498   19506.5447  force_treeallocate()/forcetree.c/4800
 118 2977 0        Nodes_base   3737.6087   23244.1534  force_treeallocate()/forcetree.c/4803
 118 2978 0     Extnodes_base   3114.6739   26358.8274  force_treeallocate()/forcetree.c/4811
 118 2979 0          Nextnode    222.3054   26581.1327  force_treeallocate()/forcetree.c/4820
 118 2980 0            Father    221.9056   26803.0383  force_treeallocate()/forcetree.c/4827
   9 2984 0      DataNodeList     58.1818   27075.4164  subfind_density()/subfind_density.c/89
   9 2985 0              Left    126.8616   27202.2780  subfind_density()/subfind_density.c/91
   9 2986 0             Right    126.8616   27329.1397  subfind_density()/subfind_density.c/92
   9 2987 0              Todo     15.8577   27344.9974  subfind_density()/subfind_density.c/93
   9 2988 0             DM_Vx    126.8616   27471.8590  subfind_density()/subfind_density.c/95
   9 2989 0             DM_Vy    126.8616   27598.7206  subfind_density()/subfind_density.c/96
   9 2990 0             DM_Vz    126.8616   27725.5823  subfind_density()/subfind_density.c/97
 118 2981 0           Ngblist     63.4012   26866.4395  subfind_density()/subfind_density.c/78
   9 2991 0    	  DensDataGet    112.1688   27837.7511  subfind_density()/subfind_density.c/161
   9 2992 0  	  DensDataResult     84.1266   27921.8777  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=9: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.1223 MB).
 118 2982 0         Dist2list    126.8023   26993.2418  subfind_density()/subfind_density.c/79
 118 2983 0    DataIndexTable     21.8182   27015.0600  subfind_density()/subfind_density.c/87
 118 2984 0      DataNodeList     58.1818   27073.2418  subfind_density()/subfind_density.c/89
task 9: endrun called with an error level of 812


 118 2985 0              Left    126.8023   27200.0441  subfind_density()/subfind_density.c/91
 118 2986 0             Right    126.8023   27326.8465  subfind_density()/subfind_density.c/92
 118 2987 0              Todo     15.8503   27342.6968  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 9
 118 2988 0             DM_Vx    126.8023   27469.4991  subfind_density()/subfind_density.c/95
 118 2989 0             DM_Vy    126.8023   27596.3014  subfind_density()/subfind_density.c/96
 118 2990 0             DM_Vz    126.8023   27723.1038  subfind_density()/subfind_density.c/97
 118 2991 0    	  DensDataGet    114.5774   27837.6812  subfind_density()/subfind_density.c/161
 118 2992 0  	  DensDataResult     85.9330   27923.6142  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=118: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.3858 MB).
task 118: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 118
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  25 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  25 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  25 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  25 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  25 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  25 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  25 2973 0         FOF_PList    380.4179   18861.2744  fof_fof()/fof.c/249
  25 2974 0         FOF_GList    634.0298   19495.3042  fof_fof()/fof.c/337
  25 2975 0             Group     10.9197   19506.2238  fof_fof()/fof.c/380
  25 2976 0   DomainNodeIndex      0.3498   19506.5737  force_treeallocate()/forcetree.c/4800
  25 2977 0        Nodes_base   3737.6087   23244.1824  force_treeallocate()/forcetree.c/4803
  25 2978 0     Extnodes_base   3114.6739   26358.8563  force_treeallocate()/forcetree.c/4811
  25 2979 0          Nextnode    222.3054   26581.1617  force_treeallocate()/forcetree.c/4820
  25 2980 0            Father    221.9056   26803.0672  force_treeallocate()/forcetree.c/4827
 102 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 102 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 102 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 102 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 102 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 102 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 102 2973 0         FOF_PList    380.1641   18861.0207  fof_fof()/fof.c/249
 102 2974 0         FOF_GList    633.6069   19494.6275  fof_fof()/fof.c/337
 102 2975 0             Group     13.9308   19508.5583  fof_fof()/fof.c/380
 102 2976 0   DomainNodeIndex      0.3498   19508.9081  force_treeallocate()/forcetree.c/4800
  25 2981 0           Ngblist     63.4030   26866.4702  subfind_density()/subfind_density.c/78
  25 2982 0         Dist2list    126.8060   26993.2762  subfind_density()/subfind_density.c/79
  25 2983 0    DataIndexTable     21.8182   27015.0944  subfind_density()/subfind_density.c/87
  25 2984 0      DataNodeList     58.1818   27073.2761  subfind_density()/subfind_density.c/89
 102 2977 0        Nodes_base   3737.6087   23246.5168  force_treeallocate()/forcetree.c/4803
 102 2978 0     Extnodes_base   3114.6739   26361.1908  force_treeallocate()/forcetree.c/4811
  25 2985 0              Left    126.8060   27200.0821  subfind_density()/subfind_density.c/91
  25 2986 0             Right    126.8060   27326.8881  subfind_density()/subfind_density.c/92
  25 2987 0              Todo     15.8507   27342.7388  subfind_density()/subfind_density.c/93
 102 2979 0          Nextnode    222.3054   26583.4961  force_treeallocate()/forcetree.c/4820
 102 2980 0            Father    221.9056   26805.4017  force_treeallocate()/forcetree.c/4827
  25 2988 0             DM_Vx    126.8060   27469.5448  subfind_density()/subfind_density.c/95
  25 2989 0             DM_Vy    126.8060   27596.3507  subfind_density()/subfind_density.c/96
 102 2981 0           Ngblist     63.3607   26868.7624  subfind_density()/subfind_density.c/78
 102 2982 0         Dist2list    126.7214   26995.4838  subfind_density()/subfind_density.c/79
 102 2983 0    DataIndexTable     21.8182   27017.3020  subfind_density()/subfind_density.c/87
  25 2990 0             DM_Vz    126.8060   27723.1567  subfind_density()/subfind_density.c/97
  25 2991 0    	  DensDataGet    108.8855   27832.0422  subfind_density()/subfind_density.c/161
  25 2992 0  	  DensDataResult     81.6641   27913.7063  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------- 102 2984 0      DataNodeList     58.1818   27075.4837  subfind_density()/subfind_density.c/89
 102 2985 0              Left    126.7214   27202.2051  subfind_density()/subfind_density.c/91
 102 2986 0             Right    126.7214   27328.9265  subfind_density()/subfind_density.c/92

 102 2987 0              Todo     15.8402   27344.7667  subfind_density()/subfind_density.c/93

Task=25: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.2937 MB).
task 25: endrun called with an error level of 812


 102 2988 0             DM_Vx    126.7214   27471.4880  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 25
 102 2989 0             DM_Vy    126.7214   27598.2094  subfind_density()/subfind_density.c/96
 102 2990 0             DM_Vz    126.7214   27724.9308  subfind_density()/subfind_density.c/97
 102 2991 0    	  DensDataGet    136.4240   27861.3548  subfind_density()/subfind_density.c/161
 102 2992 0  	  DensDataResult    102.3180   27963.6728  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=102: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=36.3272 MB).
task 102: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 102
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  99 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  99 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  99 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  99 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  99 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  99 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  99 2973 0         FOF_PList    380.5378   18861.3943  fof_fof()/fof.c/249
  99 2974 0         FOF_GList    634.2297   19495.6240  fof_fof()/fof.c/337
  99 2975 0             Group     11.3690   19506.9930  fof_fof()/fof.c/380
  99 2976 0   DomainNodeIndex      0.3498   19507.3428  force_treeallocate()/forcetree.c/4800
  99 2977 0        Nodes_base   3737.6087   23244.9515  force_treeallocate()/forcetree.c/4803
  99 2978 0     Extnodes_base   3114.6739   26359.6254  force_treeallocate()/forcetree.c/4811
  99 2979 0          Nextnode    222.3054   26581.9308  force_treeallocate()/forcetree.c/4820
  99 2980 0            Father    221.9056   26803.8363  force_treeallocate()/forcetree.c/4827
  28 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  28 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  28 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  28 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  28 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  28 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  28 2973 0         FOF_PList    380.3472   18861.2037  fof_fof()/fof.c/249
  28 2974 0         FOF_GList    633.9120   19495.1157  fof_fof()/fof.c/337
  28 2975 0             Group     11.9448   19507.0605  fof_fof()/fof.c/380
  28 2976 0   DomainNodeIndex      0.3498   19507.4103  force_treeallocate()/forcetree.c/4800
  28 2977 0        Nodes_base   3737.6087   23245.0190  force_treeallocate()/forcetree.c/4803
  28 2978 0     Extnodes_base   3114.6739   26359.6930  force_treeallocate()/forcetree.c/4811
  28 2979 0          Nextnode    222.3054   26581.9983  force_treeallocate()/forcetree.c/4820
  99 2981 0           Ngblist     63.4230   26867.2593  subfind_density()/subfind_density.c/78
  99 2982 0         Dist2list    126.8459   26994.1052  subfind_density()/subfind_density.c/79
  99 2983 0    DataIndexTable     21.8182   27015.9234  subfind_density()/subfind_density.c/87
  99 2984 0      DataNodeList     58.1818   27074.1052  subfind_density()/subfind_density.c/89
  99 2985 0              Left    126.8459   27200.9511  subfind_density()/subfind_density.c/91
  28 2980 0            Father    221.9056   26803.9039  force_treeallocate()/forcetree.c/4827
  99 2986 0             Right    126.8459   27327.7971  subfind_density()/subfind_density.c/92
  99 2987 0              Todo     15.8557   27343.6528  subfind_density()/subfind_density.c/93
  99 2988 0             DM_Vx    126.8459   27470.4988  subfind_density()/subfind_density.c/95
  28 2981 0           Ngblist     63.3912   26867.2951  subfind_density()/subfind_density.c/78
  28 2982 0         Dist2list    126.7824   26994.0775  subfind_density()/subfind_density.c/79
  99 2989 0             DM_Vy    126.8459   27597.3447  subfind_density()/subfind_density.c/96
  99 2990 0             DM_Vz    126.8459   27724.1906  subfind_density()/subfind_density.c/97
  99 2991 0    	  DensDataGet    108.6413   27832.8319  subfind_density()/subfind_density.c/161
  28 2983 0    DataIndexTable     21.8182   27015.8957  subfind_density()/subfind_density.c/87
  28 2984 0      DataNodeList     58.1818   27074.0775  subfind_density()/subfind_density.c/89
  28 2985 0              Left    126.7824   27200.8599  subfind_density()/subfind_density.c/91
  99 2992 0  	  DensDataResult     81.4810   27914.3129  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=99: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=85.6871 MB).
task 99: endrun called with an error level of 812


  28 2986 0             Right    126.7824   27327.6423  subfind_density()/subfind_density.c/92
  28 2987 0              Todo     15.8478   27343.4901  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 99
  28 2988 0             DM_Vx    126.7824   27470.2725  subfind_density()/subfind_density.c/95
  28 2989 0             DM_Vy    126.7824   27597.0549  subfind_density()/subfind_density.c/96
  28 2990 0             DM_Vz    126.7824   27723.8372  subfind_density()/subfind_density.c/97
  28 2991 0    	  DensDataGet    115.3932   27839.2305  subfind_density()/subfind_density.c/161
  28 2992 0  	  DensDataResult     86.5449   27925.7754  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=28: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.2246 MB).
task 28: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 28
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  98 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  98 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  98 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  98 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  98 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  98 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  29 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  29 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  29 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  29 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  29 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  29 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  29 2973 0         FOF_PList    380.3279   18861.1844  fof_fof()/fof.c/249
  29 2974 0         FOF_GList    633.8799   19495.0643  fof_fof()/fof.c/337
  29 2975 0             Group     11.9573   19507.0216  fof_fof()/fof.c/380
  29 2976 0   DomainNodeIndex      0.3498   19507.3714  force_treeallocate()/forcetree.c/4800
  98 2973 0         FOF_PList    380.0540   18860.9106  fof_fof()/fof.c/249
  98 2974 0         FOF_GList    633.4234   19494.3340  fof_fof()/fof.c/337
  29 2977 0        Nodes_base   3737.6087   23244.9801  force_treeallocate()/forcetree.c/4803
  29 2978 0     Extnodes_base   3114.6739   26359.6540  force_treeallocate()/forcetree.c/4811
  29 2979 0          Nextnode    222.3054   26581.9594  force_treeallocate()/forcetree.c/4820
  98 2975 0             Group     13.8281   19508.1620  fof_fof()/fof.c/380
  98 2976 0   DomainNodeIndex      0.3498   19508.5118  force_treeallocate()/forcetree.c/4800
  98 2977 0        Nodes_base   3737.6087   23246.1205  force_treeallocate()/forcetree.c/4803
  29 2980 0            Father    221.9056   26803.8650  force_treeallocate()/forcetree.c/4827
  29 2981 0           Ngblist     63.3880   26867.2530  subfind_density()/subfind_density.c/78
  29 2982 0         Dist2list    126.7760   26994.0289  subfind_density()/subfind_density.c/79
  98 2978 0     Extnodes_base   3114.6739   26360.7945  force_treeallocate()/forcetree.c/4811
  98 2979 0          Nextnode    222.3054   26583.0998  force_treeallocate()/forcetree.c/4820
  98 2980 0            Father    221.9056   26805.0054  force_treeallocate()/forcetree.c/4827
  29 2983 0    DataIndexTable     21.8182   27015.8471  subfind_density()/subfind_density.c/87
  29 2984 0      DataNodeList     58.1818   27074.0289  subfind_density()/subfind_density.c/89
  98 2981 0           Ngblist     63.3423   26868.3478  subfind_density()/subfind_density.c/78
  98 2982 0         Dist2list    126.6847   26995.0324  subfind_density()/subfind_density.c/79
  98 2983 0    DataIndexTable     21.8182   27016.8506  subfind_density()/subfind_density.c/87
  29 2985 0              Left    126.7760   27200.8049  subfind_density()/subfind_density.c/91
  29 2986 0             Right    126.7760   27327.5808  subfind_density()/subfind_density.c/92
  29 2987 0              Todo     15.8470   27343.4278  subfind_density()/subfind_density.c/93
  98 2984 0      DataNodeList     58.1818   27075.0324  subfind_density()/subfind_density.c/89
  98 2985 0              Left    126.6847   27201.7171  subfind_density()/subfind_density.c/91
  29 2988 0             DM_Vx    126.7760   27470.2038  subfind_density()/subfind_density.c/95
  29 2989 0             DM_Vy    126.7760   27596.9798  subfind_density()/subfind_density.c/96
  98 2986 0             Right    126.6847   27328.4018  subfind_density()/subfind_density.c/92
  29 2990 0             DM_Vz    126.7760   27723.7558  subfind_density()/subfind_density.c/97
  98 2987 0              Todo     15.8356   27344.2373  subfind_density()/subfind_density.c/93
  29 2991 0    	  DensDataGet    120.1196   27843.8754  subfind_density()/subfind_density.c/161
  98 2988 0             DM_Vx    126.6847   27470.9220  subfind_density()/subfind_density.c/95
  29 2992 0  	  DensDataResult     90.0897   27933.9651  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  98 2989 0             DM_Vy    126.6847   27597.6067  subfind_density()/subfind_density.c/96

  98 2990 0             DM_Vz    126.6847   27724.2914  subfind_density()/subfind_density.c/97

Task=29: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.0349 MB).
  98 2991 0    	  DensDataGet    117.2733   27841.5647  subfind_density()/subfind_density.c/161
  98 2992 0  	  DensDataResult     87.9550   27929.5197  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 29: endrun called with an error level of 812



Task=98: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.4803 MB).
task 98: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 29
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 98
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 100 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 100 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 100 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 100 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 100 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 100 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 100 2973 0         FOF_PList    380.5752   18861.4317  fof_fof()/fof.c/249
 100 2974 0         FOF_GList    634.2920   19495.7237  fof_fof()/fof.c/337
 100 2975 0             Group     12.0756   19507.7993  fof_fof()/fof.c/380
 100 2976 0   DomainNodeIndex      0.3498   19508.1491  force_treeallocate()/forcetree.c/4800
 100 2977 0        Nodes_base   3737.6087   23245.7578  force_treeallocate()/forcetree.c/4803
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 100 2978 0     Extnodes_base   3114.6739   26360.4317  force_treeallocate()/forcetree.c/4811
 100 2979 0          Nextnode    222.3054   26582.7371  force_treeallocate()/forcetree.c/4820
 100 2980 0            Father    221.9056   26804.6427  force_treeallocate()/forcetree.c/4827
  27 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  27 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  27 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  27 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  27 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  27 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  27 2973 0         FOF_PList    380.5433   18861.3998  fof_fof()/fof.c/249
  27 2974 0         FOF_GList    634.2388   19495.6386  fof_fof()/fof.c/337
  27 2975 0             Group     13.3394   19508.9780  fof_fof()/fof.c/380
  27 2976 0   DomainNodeIndex      0.3498   19509.3278  force_treeallocate()/forcetree.c/4800
  27 2977 0        Nodes_base   3737.6087   23246.9365  force_treeallocate()/forcetree.c/4803
  27 2978 0     Extnodes_base   3114.6739   26361.6104  force_treeallocate()/forcetree.c/4811
  27 2979 0          Nextnode    222.3054   26583.9158  force_treeallocate()/forcetree.c/4820
  27 2980 0            Father    221.9056   26805.8214  force_treeallocate()/forcetree.c/4827
  27 2981 0           Ngblist     63.4239   26869.2453  subfind_density()/subfind_density.c/78
  27 2982 0         Dist2list    126.8478   26996.0930  subfind_density()/subfind_density.c/79
  27 2983 0    DataIndexTable     21.8182   27017.9112  subfind_density()/subfind_density.c/87
 100 2981 0           Ngblist     63.4292   26868.0719  subfind_density()/subfind_density.c/78
 100 2982 0         Dist2list    126.8584   26994.9303  subfind_density()/subfind_density.c/79
 100 2983 0    DataIndexTable     21.8182   27016.7485  subfind_density()/subfind_density.c/87
  27 2984 0      DataNodeList     58.1818   27076.0930  subfind_density()/subfind_density.c/89
  27 2985 0              Left    126.8478   27202.9408  subfind_density()/subfind_density.c/91
  27 2986 0             Right    126.8478   27329.7885  subfind_density()/subfind_density.c/92
  27 2987 0              Todo     15.8560   27345.6445  subfind_density()/subfind_density.c/93
  27 2988 0             DM_Vx    126.8478   27472.4923  subfind_density()/subfind_density.c/95
  27 2989 0             DM_Vy    126.8478   27599.3400  subfind_density()/subfind_density.c/96
 100 2984 0      DataNodeList     58.1818   27074.9303  subfind_density()/subfind_density.c/89
 100 2985 0              Left    126.8584   27201.7887  subfind_density()/subfind_density.c/91
 100 2986 0             Right    126.8584   27328.6470  subfind_density()/subfind_density.c/92
  27 2990 0             DM_Vz    126.8478   27726.1878  subfind_density()/subfind_density.c/97
  27 2991 0    	  DensDataGet    123.8933   27850.0811  subfind_density()/subfind_density.c/161
  27 2992 0  	  DensDataResult     92.9200   27943.0011  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 100 2987 0              Todo     15.8573   27344.5043  subfind_density()/subfind_density.c/93
 100 2988 0             DM_Vx    126.8584   27471.3627  subfind_density()/subfind_density.c/95
 100 2989 0             DM_Vy    126.8584   27598.2211  subfind_density()/subfind_density.c/96

Task=27: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.9989 MB).
task 27: endrun called with an error level of 812


 100 2990 0             DM_Vz    126.8584   27725.0795  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 27
 100 2991 0    	  DensDataGet    113.8604   27838.9400  subfind_density()/subfind_density.c/161
 100 2992 0  	  DensDataResult     85.3953   27924.3353  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=100: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.6647 MB).
task 100: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 100
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 105 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 105 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 105 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 105 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 105 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 105 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 105 2973 0         FOF_PList    380.4610   18861.3175  fof_fof()/fof.c/249
 105 2974 0         FOF_GList    634.1017   19495.4192  fof_fof()/fof.c/337
 105 2975 0             Group     12.8289   19508.2481  fof_fof()/fof.c/380
 105 2976 0   DomainNodeIndex      0.3498   19508.5979  force_treeallocate()/forcetree.c/4800
 105 2977 0        Nodes_base   3737.6087   23246.2066  force_treeallocate()/forcetree.c/4803
 105 2978 0     Extnodes_base   3114.6739   26360.8805  force_treeallocate()/forcetree.c/4811
 105 2979 0          Nextnode    222.3054   26583.1859  force_treeallocate()/forcetree.c/4820
 105 2980 0            Father    221.9056   26805.0915  force_treeallocate()/forcetree.c/4827
 105 2981 0           Ngblist     63.4102   26868.5016  subfind_density()/subfind_density.c/78
 105 2982 0         Dist2list    126.8203   26995.3220  subfind_density()/subfind_density.c/79
 105 2983 0    DataIndexTable     21.8182   27017.1402  subfind_density()/subfind_density.c/87
 105 2984 0      DataNodeList     58.1818   27075.3220  subfind_density()/subfind_density.c/89
 105 2985 0              Left    126.8203   27202.1423  subfind_density()/subfind_density.c/91
 105 2986 0             Right    126.8203   27328.9626  subfind_density()/subfind_density.c/92
 105 2987 0              Todo     15.8525   27344.8152  subfind_density()/subfind_density.c/93
 105 2988 0             DM_Vx    126.8203   27471.6355  subfind_density()/subfind_density.c/95
 105 2989 0             DM_Vy    126.8203   27598.4558  subfind_density()/subfind_density.c/96
 105 2990 0             DM_Vz    126.8203   27725.2762  subfind_density()/subfind_density.c/97
 105 2991 0    	  DensDataGet    114.4182   27839.6944  subfind_density()/subfind_density.c/161
 105 2992 0  	  DensDataResult     85.8137   27925.5080  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=105: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.492 MB).
task 105: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 105
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  12 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  12 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  12 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  12 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  12 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  12 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  12 2973 0         FOF_PList    380.4461   18861.3026  fof_fof()/fof.c/249
  12 2974 0         FOF_GList    634.0768   19495.3794  fof_fof()/fof.c/337
  12 2975 0             Group     10.9197   19506.2990  fof_fof()/fof.c/380
  12 2976 0   DomainNodeIndex      0.3498   19506.6489  force_treeallocate()/forcetree.c/4800
  12 2977 0        Nodes_base   3737.6087   23244.2576  force_treeallocate()/forcetree.c/4803
  12 2978 0     Extnodes_base   3114.6739   26358.9315  force_treeallocate()/forcetree.c/4811
  12 2979 0          Nextnode    222.3054   26581.2369  force_treeallocate()/forcetree.c/4820
  12 2980 0            Father    221.9056   26803.1424  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  12 2981 0           Ngblist     63.4077   26866.5501  subfind_density()/subfind_density.c/78
  12 2982 0         Dist2list    126.8154   26993.3655  subfind_density()/subfind_density.c/79
  12 2983 0    DataIndexTable     21.8182   27015.1836  subfind_density()/subfind_density.c/87
   8 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   8 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   8 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 119 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 119 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 119 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 119 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 119 2973 0         FOF_PList    380.1763   18861.0328  fof_fof()/fof.c/249
 119 2974 0         FOF_GList    633.6271   19494.6599  fof_fof()/fof.c/337
 119 2975 0             Group     11.5993   19506.2592  fof_fof()/fof.c/380
 119 2976 0   DomainNodeIndex      0.3498   19506.6091  force_treeallocate()/forcetree.c/4800
 119 2977 0        Nodes_base   3737.6087   23244.2178  force_treeallocate()/forcetree.c/4803
 119 2978 0     Extnodes_base   3114.6739   26358.8917  force_treeallocate()/forcetree.c/4811
 119 2979 0          Nextnode    222.3054   26581.1971  force_treeallocate()/forcetree.c/4820
 119 2980 0            Father    221.9056   26803.1026  force_treeallocate()/forcetree.c/4827
  12 2984 0      DataNodeList     58.1818   27073.3654  subfind_density()/subfind_density.c/89
  12 2985 0              Left    126.8154   27200.1808  subfind_density()/subfind_density.c/91
  12 2986 0             Right    126.8154   27326.9961  subfind_density()/subfind_density.c/92
   8 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   8 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 119 2981 0           Ngblist     63.3627   26866.4654  subfind_density()/subfind_density.c/78
 119 2982 0         Dist2list    126.7254   26993.1908  subfind_density()/subfind_density.c/79
 119 2983 0    DataIndexTable     21.8182   27015.0090  subfind_density()/subfind_density.c/87
 119 2984 0      DataNodeList     58.1818   27073.1908  subfind_density()/subfind_density.c/89
  12 2987 0              Todo     15.8519   27342.8481  subfind_density()/subfind_density.c/93
  12 2988 0             DM_Vx    126.8154   27469.6634  subfind_density()/subfind_density.c/95
  12 2989 0             DM_Vy    126.8154   27596.4788  subfind_density()/subfind_density.c/96
   8 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   8 2973 0         FOF_PList    380.5701   18861.4267  fof_fof()/fof.c/249
   8 2974 0         FOF_GList    634.2835   19495.7102  fof_fof()/fof.c/337
 119 2985 0              Left    126.7254   27199.9162  subfind_density()/subfind_density.c/91
 119 2986 0             Right    126.7254   27326.6416  subfind_density()/subfind_density.c/92
 119 2987 0              Todo     15.8407   27342.4823  subfind_density()/subfind_density.c/93
 119 2988 0             DM_Vx    126.7254   27469.2077  subfind_density()/subfind_density.c/95
  12 2990 0             DM_Vz    126.8154   27723.2941  subfind_density()/subfind_density.c/97
   8 2975 0             Group     12.3090   19508.0192  fof_fof()/fof.c/380
   8 2976 0   DomainNodeIndex      0.3498   19508.3690  force_treeallocate()/forcetree.c/4800
   8 2977 0        Nodes_base   3737.6087   23245.9777  force_treeallocate()/forcetree.c/4803
 119 2989 0             DM_Vy    126.7254   27595.9331  subfind_density()/subfind_density.c/96
 119 2990 0             DM_Vz    126.7254   27722.6586  subfind_density()/subfind_density.c/97
 119 2991 0    	  DensDataGet    116.9057   27839.5643  subfind_density()/subfind_density.c/161
  12 2991 0    	  DensDataGet    116.9379   27840.2320  subfind_density()/subfind_density.c/161
   8 2978 0     Extnodes_base   3114.6739   26360.6516  force_treeallocate()/forcetree.c/4811
   8 2979 0          Nextnode    222.3054   26582.9570  force_treeallocate()/forcetree.c/4820
 119 2992 0  	  DensDataResult     87.6793   27927.2435  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=119: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.7565 MB).
task 119: endrun called with an error level of 812


  12 2992 0  	  DensDataResult     87.7034   27927.9354  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   8 2980 0            Father    221.9056   26804.8626  force_treeallocate()/forcetree.c/4827
   8 2981 0           Ngblist     63.4284   26868.2909  subfind_density()/subfind_density.c/78
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 119

Task=12: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.0646 MB).
task 12: endrun called with an error level of 812


   8 2982 0         Dist2list    126.8567   26995.1477  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 12
   8 2983 0    DataIndexTable     21.8182   27016.9658  subfind_density()/subfind_density.c/87
  14 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  14 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  14 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  14 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  14 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  14 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  14 2973 0         FOF_PList    380.6234   18861.4800  fof_fof()/fof.c/249
  14 2974 0         FOF_GList    634.3724   19495.8523  fof_fof()/fof.c/337
  14 2975 0             Group     11.4084   19507.2607  fof_fof()/fof.c/380
  14 2976 0   DomainNodeIndex      0.3498   19507.6105  force_treeallocate()/forcetree.c/4800
  14 2977 0        Nodes_base   3737.6087   23245.2192  force_treeallocate()/forcetree.c/4803
  14 2978 0     Extnodes_base   3114.6739   26359.8932  force_treeallocate()/forcetree.c/4811
 113 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 113 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 113 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 113 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 113 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 113 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 113 2973 0         FOF_PList    380.4217   18861.2782  fof_fof()/fof.c/249
 113 2974 0         FOF_GList    634.0362   19495.3144  fof_fof()/fof.c/337
 113 2975 0             Group     12.9150   19508.2294  fof_fof()/fof.c/380
 113 2976 0   DomainNodeIndex      0.3498   19508.5792  force_treeallocate()/forcetree.c/4800
 113 2977 0        Nodes_base   3737.6087   23246.1879  force_treeallocate()/forcetree.c/4803
 113 2978 0     Extnodes_base   3114.6739   26360.8618  force_treeallocate()/forcetree.c/4811
 113 2979 0          Nextnode    222.3054   26583.1672  force_treeallocate()/forcetree.c/4820
 113 2980 0            Father    221.9056   26805.0728  force_treeallocate()/forcetree.c/4827
   8 2984 0      DataNodeList     58.1818   27075.1476  subfind_density()/subfind_density.c/89
  14 2979 0          Nextnode    222.3054   26582.1985  force_treeallocate()/forcetree.c/4820
  14 2980 0            Father    221.9056   26804.1041  force_treeallocate()/forcetree.c/4827
  14 2981 0           Ngblist     63.4372   26867.5414  subfind_density()/subfind_density.c/78
 113 2981 0           Ngblist     63.4036   26868.4764  subfind_density()/subfind_density.c/78
 113 2982 0         Dist2list    126.8072   26995.2837  subfind_density()/subfind_density.c/79
 113 2983 0    DataIndexTable     21.8182   27017.1018  subfind_density()/subfind_density.c/87
 113 2984 0      DataNodeList     58.1818   27075.2836  subfind_density()/subfind_density.c/89
 113 2985 0              Left    126.8072   27202.0909  subfind_density()/subfind_density.c/91
   8 2985 0              Left    126.8567   27202.0043  subfind_density()/subfind_density.c/91
   8 2986 0             Right    126.8567   27328.8610  subfind_density()/subfind_density.c/92
  14 2982 0         Dist2list    126.8745   26994.4158  subfind_density()/subfind_density.c/79
  14 2983 0    DataIndexTable     21.8182   27016.2340  subfind_density()/subfind_density.c/87
  14 2984 0      DataNodeList     58.1818   27074.4158  subfind_density()/subfind_density.c/89
 113 2986 0             Right    126.8072   27328.8981  subfind_density()/subfind_density.c/92
 113 2987 0              Todo     15.8509   27344.7490  subfind_density()/subfind_density.c/93
 113 2988 0             DM_Vx    126.8072   27471.5562  subfind_density()/subfind_density.c/95
   8 2987 0              Todo     15.8571   27344.7181  subfind_density()/subfind_density.c/93
  14 2985 0              Left    126.8745   27201.2903  subfind_density()/subfind_density.c/91
  14 2986 0             Right    126.8745   27328.1647  subfind_density()/subfind_density.c/92
  14 2987 0              Todo     15.8593   27344.0241  subfind_density()/subfind_density.c/93
 113 2989 0             DM_Vy    126.8072   27598.3635  subfind_density()/subfind_density.c/96
 113 2990 0             DM_Vz    126.8072   27725.1707  subfind_density()/subfind_density.c/97
 113 2991 0    	  DensDataGet    114.9730   27840.1437  subfind_density()/subfind_density.c/161
   8 2988 0             DM_Vx    126.8567   27471.5748  subfind_density()/subfind_density.c/95
  14 2988 0             DM_Vx    126.8745   27470.8985  subfind_density()/subfind_density.c/95
  14 2989 0             DM_Vy    126.8745   27597.7730  subfind_density()/subfind_density.c/96
  14 2990 0             DM_Vz    126.8745   27724.6475  subfind_density()/subfind_density.c/97
 113 2992 0  	  DensDataResult     86.2298   27926.3735  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=113: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.6265 MB).
task 113: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   8 2989 0             DM_Vy    126.8567   27598.4315  subfind_density()/subfind_density.c/96
  14 2991 0    	  DensDataGet    116.6427   27841.2902  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 113
  10 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  10 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  10 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  10 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  10 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  10 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  10 2973 0         FOF_PList    380.2566   18861.1131  fof_fof()/fof.c/249
  10 2974 0         FOF_GList    633.7609   19494.8741  fof_fof()/fof.c/337
  10 2975 0             Group     14.0854   19508.9594  fof_fof()/fof.c/380
 117 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 117 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 117 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 117 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 117 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 117 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 117 2973 0         FOF_PList    379.7723   18860.6289  fof_fof()/fof.c/249
 117 2974 0         FOF_GList    632.9539   19493.5827  fof_fof()/fof.c/337
 117 2975 0             Group     12.4802   19506.0630  fof_fof()/fof.c/380
 117 2976 0   DomainNodeIndex      0.3498   19506.4128  force_treeallocate()/forcetree.c/4800
 117 2977 0        Nodes_base   3737.6087   23244.0215  force_treeallocate()/forcetree.c/4803
 117 2978 0     Extnodes_base   3114.6739   26358.6954  force_treeallocate()/forcetree.c/4811
   8 2990 0             DM_Vz    126.8567   27725.2882  subfind_density()/subfind_density.c/97
   8 2991 0    	  DensDataGet    120.2492   27845.5374  subfind_density()/subfind_density.c/161
  14 2992 0  	  DensDataResult     87.4820   27928.7722  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  10 2976 0   DomainNodeIndex      0.3498   19509.3093  force_treeallocate()/forcetree.c/4800
  10 2977 0        Nodes_base   3737.6087   23246.9180  force_treeallocate()/forcetree.c/4803
 117 2979 0          Nextnode    222.3054   26581.0008  force_treeallocate()/forcetree.c/4820
 117 2980 0            Father    221.9056   26802.9064  force_treeallocate()/forcetree.c/4827
 117 2981 0           Ngblist     63.2954   26866.2017  subfind_density()/subfind_density.c/78
 117 2982 0         Dist2list    126.5908   26992.7925  subfind_density()/subfind_density.c/79
   8 2992 0  	  DensDataResult     90.1869   27935.7243  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

  10 2978 0     Extnodes_base   3114.6739   26361.5919  force_treeallocate()/forcetree.c/4811
  10 2979 0          Nextnode    222.3054   26583.8973  force_treeallocate()/forcetree.c/4820
  10 2980 0            Father    221.9056   26805.8028  force_treeallocate()/forcetree.c/4827
 117 2983 0    DataIndexTable     21.8182   27014.6107  subfind_density()/subfind_density.c/87
 117 2984 0      DataNodeList     58.1818   27072.7925  subfind_density()/subfind_density.c/89
 117 2985 0              Left    126.5908   27199.3833  subfind_density()/subfind_density.c/91
 117 2986 0             Right    126.5908   27325.9740  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   1 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   1 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   1 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   1 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61

Task=8: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.2757 MB).
task 8: endrun called with an error level of 812



Task=14: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.2278 MB).
task 14: endrun called with an error level of 812


  10 2981 0           Ngblist     63.3761   26869.1789  subfind_density()/subfind_density.c/78
  10 2982 0         Dist2list    126.7522   26995.9311  subfind_density()/subfind_density.c/79
  10 2983 0    DataIndexTable     21.8182   27017.7493  subfind_density()/subfind_density.c/87
 117 2987 0              Todo     15.8239   27341.7979  subfind_density()/subfind_density.c/93
 117 2988 0             DM_Vx    126.5908   27468.3887  subfind_density()/subfind_density.c/95
   1 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   1 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   1 2973 0         FOF_PList    380.3675   18861.2240  fof_fof()/fof.c/249
   1 2974 0         FOF_GList    633.9458   19495.1698  fof_fof()/fof.c/337
   1 2975 0             Group     10.9197   19506.0895  fof_fof()/fof.c/380
   1 2976 0   DomainNodeIndex      0.3498   19506.4393  force_treeallocate()/forcetree.c/4800
   1 2977 0        Nodes_base   3737.6087   23244.0480  force_treeallocate()/forcetree.c/4803
   1 2978 0     Extnodes_base   3114.6739   26358.7219  force_treeallocate()/forcetree.c/4811
   1 2979 0          Nextnode    222.3054   26581.0273  force_treeallocate()/forcetree.c/4820
   1 2980 0            Father    221.9056   26802.9329  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 8
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 14
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  10 2984 0      DataNodeList     58.1818   27075.9311  subfind_density()/subfind_density.c/89
  10 2985 0              Left    126.7522   27202.6833  subfind_density()/subfind_density.c/91
  10 2986 0             Right    126.7522   27329.4355  subfind_density()/subfind_density.c/92
 117 2989 0             DM_Vy    126.5908   27594.9794  subfind_density()/subfind_density.c/96
 117 2990 0             DM_Vz    126.5908   27721.5702  subfind_density()/subfind_density.c/97
 117 2991 0    	  DensDataGet    123.8373   27845.4076  subfind_density()/subfind_density.c/161
   1 2981 0           Ngblist     63.3946   26866.3275  subfind_density()/subfind_density.c/78
  17 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  17 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  17 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  17 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  17 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  17 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  17 2973 0         FOF_PList    380.3446   18861.2011  fof_fof()/fof.c/249
  17 2974 0         FOF_GList    633.9076   19495.1087  fof_fof()/fof.c/337
  17 2975 0             Group     10.9197   19506.0284  fof_fof()/fof.c/380
  17 2976 0   DomainNodeIndex      0.3498   19506.3782  force_treeallocate()/forcetree.c/4800
  17 2977 0        Nodes_base   3737.6087   23243.9869  force_treeallocate()/forcetree.c/4803
 An HDF5 error was detected, we'd better stop.
 110 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 110 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 110 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 110 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 110 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 110 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 110 2973 0         FOF_PList    380.5600   18861.4166  fof_fof()/fof.c/249
 110 2974 0         FOF_GList    634.2667   19495.6833  fof_fof()/fof.c/337
 110 2975 0             Group     12.9108   19508.5941  fof_fof()/fof.c/380
 110 2976 0   DomainNodeIndex      0.3498   19508.9440  force_treeallocate()/forcetree.c/4800
 110 2977 0        Nodes_base   3737.6087   23246.5527  force_treeallocate()/forcetree.c/4803
 110 2978 0     Extnodes_base   3114.6739   26361.2266  force_treeallocate()/forcetree.c/4811
 110 2979 0          Nextnode    222.3054   26583.5320  force_treeallocate()/forcetree.c/4820
 126 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 126 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 126 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 126 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 126 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 126 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 126 2973 0         FOF_PList    380.4377   18861.2942  fof_fof()/fof.c/249
 126 2974 0         FOF_GList    634.0628   19495.3570  fof_fof()/fof.c/337
 126 2975 0             Group     11.5215   19506.8785  fof_fof()/fof.c/380
 126 2976 0   DomainNodeIndex      0.3498   19507.2283  force_treeallocate()/forcetree.c/4800
 126 2977 0        Nodes_base   3737.6087   23244.8370  force_treeallocate()/forcetree.c/4803
 126 2978 0     Extnodes_base   3114.6739   26359.5109  force_treeallocate()/forcetree.c/4811
  10 2987 0              Todo     15.8440   27345.2795  subfind_density()/subfind_density.c/93
 117 2992 0  	  DensDataResult     92.8780   27938.2856  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   1 2982 0         Dist2list    126.7892   26993.1167  subfind_density()/subfind_density.c/79
   1 2983 0    DataIndexTable     21.8182   27014.9348  subfind_density()/subfind_density.c/87
  17 2978 0     Extnodes_base   3114.6739   26358.6609  force_treeallocate()/forcetree.c/4811
  17 2979 0          Nextnode    222.3054   26580.9662  force_treeallocate()/forcetree.c/4820
  17 2980 0            Father    221.9056   26802.8718  force_treeallocate()/forcetree.c/4827
  17 2981 0           Ngblist     63.3908   26866.2626  subfind_density()/subfind_density.c/78
  17 2982 0         Dist2list    126.7815   26993.0441  subfind_density()/subfind_density.c/79
HDF5-DIAG: Error detected in HDF5 (1.8.12)  110 2980 0            Father    221.9056   26805.4375  force_treeallocate()/forcetree.c/4827
 126 2979 0          Nextnode    222.3054   26581.8163  force_treeallocate()/forcetree.c/4820
 126 2980 0            Father    221.9056   26803.7219  force_treeallocate()/forcetree.c/4827
 126 2981 0           Ngblist     63.4063   26867.1282  subfind_density()/subfind_density.c/78
 126 2982 0         Dist2list    126.8126   26993.9408  subfind_density()/subfind_density.c/79
  10 2988 0             DM_Vx    126.7522   27472.0317  subfind_density()/subfind_density.c/95

Task=117: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.7144 MB).
task 117: endrun called with an error level of 812


   1 2984 0      DataNodeList     58.1818   27073.1166  subfind_density()/subfind_density.c/89
  17 2983 0    DataIndexTable     21.8182   27014.8623  subfind_density()/subfind_density.c/87
  17 2984 0      DataNodeList     58.1818   27073.0441  subfind_density()/subfind_density.c/89
  17 2985 0              Left    126.7815   27199.8256  subfind_density()/subfind_density.c/91
MPI-process 8 110 2981 0           Ngblist     63.4267   26868.8642  subfind_density()/subfind_density.c/78
 110 2982 0         Dist2list    126.8533   26995.7176  subfind_density()/subfind_density.c/79
 110 2983 0    DataIndexTable     21.8182   27017.5357  subfind_density()/subfind_density.c/87
 126 2983 0    DataIndexTable     21.8182   27015.7589  subfind_density()/subfind_density.c/87
 126 2984 0      DataNodeList     58.1818   27073.9407  subfind_density()/subfind_density.c/89
 126 2985 0              Left    126.8126   27200.7533  subfind_density()/subfind_density.c/91
  10 2989 0             DM_Vy    126.7522   27598.7839  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 117
   1 2985 0              Left    126.7892   27199.9058  subfind_density()/subfind_density.c/91
  17 2986 0             Right    126.7815   27326.6071  subfind_density()/subfind_density.c/92
  17 2987 0              Todo     15.8477   27342.4548  subfind_density()/subfind_density.c/93
  17 2988 0             DM_Vx    126.7815   27469.2363  subfind_density()/subfind_density.c/95
  17 2989 0             DM_Vy    126.7815   27596.0179  subfind_density()/subfind_density.c/96
:
 110 2984 0      DataNodeList     58.1818   27075.7175  subfind_density()/subfind_density.c/89
 110 2985 0              Left    126.8533   27202.5709  subfind_density()/subfind_density.c/91
 110 2986 0             Right    126.8533   27329.4242  subfind_density()/subfind_density.c/92
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  10 2990 0             DM_Vz    126.7522   27725.5361  subfind_density()/subfind_density.c/97
   1 2986 0             Right    126.7892   27326.6949  subfind_density()/subfind_density.c/92
  17 2990 0             DM_Vz    126.7815   27722.7994  subfind_density()/subfind_density.c/97
  17 2991 0    	  DensDataGet    124.2425   27847.0419  subfind_density()/subfind_density.c/161
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
 110 2987 0              Todo     15.8567   27345.2809  subfind_density()/subfind_density.c/93
 110 2988 0             DM_Vx    126.8533   27472.1343  subfind_density()/subfind_density.c/95
 126 2986 0             Right    126.8126   27327.5658  subfind_density()/subfind_density.c/92
 126 2987 0              Todo     15.8516   27343.4174  subfind_density()/subfind_density.c/93
 126 2988 0             DM_Vx    126.8126   27470.2300  subfind_density()/subfind_density.c/95
 126 2989 0             DM_Vy    126.8126   27597.0425  subfind_density()/subfind_density.c/96
 126 2990 0             DM_Vz    126.8126   27723.8551  subfind_density()/subfind_density.c/97
 126 2991 0    	  DensDataGet    118.5103   27842.3653  subfind_density()/subfind_density.c/161
  10 2991 0    	  DensDataGet    118.1147   27843.6508  subfind_density()/subfind_density.c/161
   1 2987 0              Todo     15.8486   27342.5436  subfind_density()/subfind_density.c/93
   1 2988 0             DM_Vx    126.7892   27469.3328  subfind_density()/subfind_density.c/95
  17 2992 0  	  DensDataResult     93.1819   27940.2237  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
task 8: endrun called with an error level of 124


 110 2989 0             DM_Vy    126.8533   27598.9876  subfind_density()/subfind_density.c/96
 126 2992 0  	  DensDataResult     88.8827   27931.2480  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  10 2992 0  	  DensDataResult     88.5860   27932.2368  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------   1 2989 0             DM_Vy    126.7892   27596.1219  subfind_density()/subfind_density.c/96

Task=17: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.7763 MB).
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 8
 110 2990 0             DM_Vz    126.8533   27725.8410  subfind_density()/subfind_density.c/97


   1 2990 0             DM_Vz    126.7892   27722.9111  subfind_density()/subfind_density.c/97
   1 2991 0    	  DensDataGet    127.4280   27850.3391  subfind_density()/subfind_density.c/161
task 17: endrun called with an error level of 812


 110 2991 0    	  DensDataGet    135.5414   27861.3823  subfind_density()/subfind_density.c/161

Task=126: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.752 MB).
task 126: endrun called with an error level of 812



Task=10: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.7632 MB).
   1 2992 0  	  DensDataResult     95.5710   27945.9101  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 17
 110 2992 0  	  DensDataResult    101.6560   27963.0384  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 127 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 127 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 127 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 127 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 127 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 127 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 127 2973 0         FOF_PList    380.6482   18861.5047  fof_fof()/fof.c/249
 127 2974 0         FOF_GList    634.4137   19495.9184  fof_fof()/fof.c/337
 127 2975 0             Group     11.8722   19507.7906  fof_fof()/fof.c/380
 127 2976 0   DomainNodeIndex      0.3498   19508.1404  force_treeallocate()/forcetree.c/4800
 127 2977 0        Nodes_base   3737.6087   23245.7491  force_treeallocate()/forcetree.c/4803
 127 2978 0     Extnodes_base   3114.6739   26360.4231  force_treeallocate()/forcetree.c/4811
 127 2979 0          Nextnode    222.3054   26582.7284  force_treeallocate()/forcetree.c/4820
 127 2980 0            Father    221.9056   26804.6340  force_treeallocate()/forcetree.c/4827
 127 2981 0           Ngblist     63.4414   26868.0754  subfind_density()/subfind_density.c/78
 127 2982 0         Dist2list    126.8827   26994.9581  subfind_density()/subfind_density.c/79
task 10: endrun called with an error level of 812



Task=1: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.0899 MB).
task 1: endrun called with an error level of 812



Task=110: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=36.9616 MB).
task 110: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 126
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 10
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 110
 127 2983 0    DataIndexTable     21.8182   27016.7763  subfind_density()/subfind_density.c/87
 127 2984 0      DataNodeList     58.1818   27074.9581  subfind_density()/subfind_density.c/89
 127 2985 0              Left    126.8827   27201.8408  subfind_density()/subfind_density.c/91
 127 2986 0             Right    126.8827   27328.7236  subfind_density()/subfind_density.c/92
 127 2987 0              Todo     15.8603   27344.5839  subfind_density()/subfind_density.c/93
 127 2988 0             DM_Vx    126.8827   27471.4666  subfind_density()/subfind_density.c/95
 127 2989 0             DM_Vy    126.8827   27598.3494  subfind_density()/subfind_density.c/96
 127 2990 0             DM_Vz    126.8827   27725.2321  subfind_density()/subfind_density.c/97
 127 2991 0    	  DensDataGet    124.2330   27849.4651  subfind_density()/subfind_density.c/161
 127 2992 0  	  DensDataResult     93.1747   27942.6398  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=127: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.3602 MB).
task 127: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 127
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  23 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  23 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  23 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  23 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  23 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  23 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  23 2973 0         FOF_PList    380.5180   18861.3745  fof_fof()/fof.c/249
  23 2974 0         FOF_GList    634.1966   19495.5711  fof_fof()/fof.c/337
  23 2975 0             Group     12.0974   19507.6684  fof_fof()/fof.c/380
  23 2976 0   DomainNodeIndex      0.3498   19508.0182  force_treeallocate()/forcetree.c/4800
  23 2977 0        Nodes_base   3737.6087   23245.6269  force_treeallocate()/forcetree.c/4803
  23 2978 0     Extnodes_base   3114.6739   26360.3009  force_treeallocate()/forcetree.c/4811
  23 2979 0          Nextnode    222.3054   26582.6062  force_treeallocate()/forcetree.c/4820
  23 2980 0            Father    221.9056   26804.5118  force_treeallocate()/forcetree.c/4827
  23 2981 0           Ngblist     63.4197   26867.9315  subfind_density()/subfind_density.c/78
  23 2982 0         Dist2list    126.8393   26994.7708  subfind_density()/subfind_density.c/79
  23 2983 0    DataIndexTable     21.8182   27016.5890  subfind_density()/subfind_density.c/87
  23 2984 0      DataNodeList     58.1818   27074.7708  subfind_density()/subfind_density.c/89
  23 2985 0              Left    126.8393   27201.6101  subfind_density()/subfind_density.c/91
  23 2986 0             Right    126.8393   27328.4494  subfind_density()/subfind_density.c/92
  23 2987 0              Todo     15.8549   27344.3043  subfind_density()/subfind_density.c/93
  23 2988 0             DM_Vx    126.8393   27471.1436  subfind_density()/subfind_density.c/95
  23 2989 0             DM_Vy    126.8393   27597.9830  subfind_density()/subfind_density.c/96
  23 2990 0             DM_Vz    126.8393   27724.8223  subfind_density()/subfind_density.c/97
  23 2991 0    	  DensDataGet    143.2136   27868.0358  subfind_density()/subfind_density.c/161
  23 2992 0  	  DensDataResult    107.4102   27975.4460  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=23: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=24.554 MB).
task 23: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 23
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  13 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  13 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  13 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  13 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  13 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  13 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  13 2973 0         FOF_PList    380.4099   18861.2664  fof_fof()/fof.c/249
  13 2974 0         FOF_GList    634.0164   19495.2828  fof_fof()/fof.c/337
  13 2975 0             Group     13.3726   19508.6554  fof_fof()/fof.c/380
  13 2976 0   DomainNodeIndex      0.3498   19509.0052  force_treeallocate()/forcetree.c/4800
  13 2977 0        Nodes_base   3737.6087   23246.6139  force_treeallocate()/forcetree.c/4803
  13 2978 0     Extnodes_base   3114.6739   26361.2878  force_treeallocate()/forcetree.c/4811
  13 2979 0          Nextnode    222.3054   26583.5932  force_treeallocate()/forcetree.c/4820
 114 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 114 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 114 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 114 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 114 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 114 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 114 2973 0         FOF_PList    379.9795   18860.8360  fof_fof()/fof.c/249
 114 2974 0         FOF_GList    633.2991   19494.1351  fof_fof()/fof.c/337
 114 2975 0             Group     12.3858   19506.5209  fof_fof()/fof.c/380
  13 2980 0            Father    221.9056   26805.4988  force_treeallocate()/forcetree.c/4827
  13 2981 0           Ngblist     63.4016   26868.9004  subfind_density()/subfind_density.c/78
  13 2982 0         Dist2list    126.8033   26995.7037  subfind_density()/subfind_density.c/79
  13 2983 0    DataIndexTable     21.8182   27017.5219  subfind_density()/subfind_density.c/87
 114 2976 0   DomainNodeIndex      0.3498   19506.8707  force_treeallocate()/forcetree.c/4800
 114 2977 0        Nodes_base   3737.6087   23244.4794  force_treeallocate()/forcetree.c/4803
  13 2984 0      DataNodeList     58.1818   27075.7037  subfind_density()/subfind_density.c/89
  13 2985 0              Left    126.8033   27202.5069  subfind_density()/subfind_density.c/91
  13 2986 0             Right    126.8033   27329.3102  subfind_density()/subfind_density.c/92
 114 2978 0     Extnodes_base   3114.6739   26359.1533  force_treeallocate()/forcetree.c/4811
 114 2979 0          Nextnode    222.3054   26581.4587  force_treeallocate()/forcetree.c/4820
 114 2980 0            Father    221.9056   26803.3643  force_treeallocate()/forcetree.c/4827
 114 2981 0           Ngblist     63.3299   26866.6942  subfind_density()/subfind_density.c/78
  13 2987 0              Todo     15.8504   27345.1606  subfind_density()/subfind_density.c/93
  13 2988 0             DM_Vx    126.8033   27471.9639  subfind_density()/subfind_density.c/95
  13 2989 0             DM_Vy    126.8033   27598.7672  subfind_density()/subfind_density.c/96
 114 2982 0         Dist2list    126.6598   26993.3540  subfind_density()/subfind_density.c/79
 114 2983 0    DataIndexTable     21.8182   27015.1722  subfind_density()/subfind_density.c/87
 114 2984 0      DataNodeList     58.1818   27073.3540  subfind_density()/subfind_density.c/89
  13 2990 0             DM_Vz    126.8033   27725.5705  subfind_density()/subfind_density.c/97
  13 2991 0    	  DensDataGet    110.0055   27835.5760  subfind_density()/subfind_density.c/161
  13 2992 0  	  DensDataResult     82.5041   27918.0801  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 114 2985 0              Left    126.6598   27200.0138  subfind_density()/subfind_density.c/91
 114 2986 0             Right    126.6598   27326.6736  subfind_density()/subfind_density.c/92
 114 2987 0              Todo     15.8325   27342.5061  subfind_density()/subfind_density.c/93

Task=13: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.9199 MB).
task 13: endrun called with an error level of 812


 114 2988 0             DM_Vx    126.6598   27469.1659  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 13
 114 2989 0             DM_Vy    126.6598   27595.8258  subfind_density()/subfind_density.c/96
 114 2990 0             DM_Vz    126.6598   27722.4856  subfind_density()/subfind_density.c/97
 114 2991 0    	  DensDataGet    113.4015   27835.8871  subfind_density()/subfind_density.c/161
 114 2992 0  	  DensDataResult     85.0511   27920.9382  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=114: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.0618 MB).
task 114: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 114
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  30 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  30 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  30 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  30 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  30 2973 0         FOF_PList    380.3635   18861.2201  fof_fof()/fof.c/249
  30 2974 0         FOF_GList    633.9392   19495.1593  fof_fof()/fof.c/337
  30 2975 0             Group     12.6545   19507.8139  fof_fof()/fof.c/380
  30 2976 0   DomainNodeIndex      0.3498   19508.1637  force_treeallocate()/forcetree.c/4800
  30 2977 0        Nodes_base   3737.6087   23245.7724  force_treeallocate()/forcetree.c/4803
  30 2978 0     Extnodes_base   3114.6739   26360.4463  force_treeallocate()/forcetree.c/4811
  30 2979 0          Nextnode    222.3054   26582.7517  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  30 2980 0            Father    221.9056   26804.6573  force_treeallocate()/forcetree.c/4827
  30 2981 0           Ngblist     63.3939   26868.0512  subfind_density()/subfind_density.c/78
  30 2982 0         Dist2list    126.7878   26994.8391  subfind_density()/subfind_density.c/79
  30 2983 0    DataIndexTable     21.8182   27016.6572  subfind_density()/subfind_density.c/87
  16 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  16 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  16 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  16 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  16 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  16 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 111 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 111 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 111 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 111 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 111 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 111 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 111 2973 0         FOF_PList    380.1750   18861.0315  fof_fof()/fof.c/249
 111 2974 0         FOF_GList    633.6250   19494.6565  fof_fof()/fof.c/337
 111 2975 0             Group     11.8898   19506.5464  fof_fof()/fof.c/380
 111 2976 0   DomainNodeIndex      0.3498   19506.8962  force_treeallocate()/forcetree.c/4800
 111 2977 0        Nodes_base   3737.6087   23244.5049  force_treeallocate()/forcetree.c/4803
 111 2978 0     Extnodes_base   3114.6739   26359.1788  force_treeallocate()/forcetree.c/4811
 111 2979 0          Nextnode    222.3054   26581.4842  force_treeallocate()/forcetree.c/4820
  97 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  97 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  97 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  97 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  97 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  97 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  97 2973 0         FOF_PList    380.1765   18861.0330  fof_fof()/fof.c/249
  97 2974 0         FOF_GList    633.6274   19494.6604  fof_fof()/fof.c/337
  97 2975 0             Group     11.6408   19506.3012  fof_fof()/fof.c/380
  97 2976 0   DomainNodeIndex      0.3498   19506.6511  force_treeallocate()/forcetree.c/4800
  97 2977 0        Nodes_base   3737.6087   23244.2598  force_treeallocate()/forcetree.c/4803
  97 2978 0     Extnodes_base   3114.6739   26358.9337  force_treeallocate()/forcetree.c/4811
  97 2979 0          Nextnode    222.3054   26581.2391  force_treeallocate()/forcetree.c/4820
  97 2980 0            Father    221.9056   26803.1446  force_treeallocate()/forcetree.c/4827
  97 2981 0           Ngblist     63.3627   26866.5074  subfind_density()/subfind_density.c/78
  97 2982 0         Dist2list    126.7255   26993.2329  subfind_density()/subfind_density.c/79
  97 2983 0    DataIndexTable     21.8182   27015.0510  subfind_density()/subfind_density.c/87
  97 2984 0      DataNodeList     58.1818   27073.2328  subfind_density()/subfind_density.c/89
  97 2985 0              Left    126.7255   27199.9583  subfind_density()/subfind_density.c/91
  30 2984 0      DataNodeList     58.1818   27074.8390  subfind_density()/subfind_density.c/89
  30 2985 0              Left    126.7878   27201.6269  subfind_density()/subfind_density.c/91
  30 2986 0             Right    126.7878   27328.4147  subfind_density()/subfind_density.c/92
  16 2973 0         FOF_PList    380.5805   18861.4370  fof_fof()/fof.c/249
  16 2974 0         FOF_GList    634.3008   19495.7378  fof_fof()/fof.c/337
  16 2975 0             Group     11.6232   19507.3609  fof_fof()/fof.c/380
 111 2980 0            Father    221.9056   26803.3898  force_treeallocate()/forcetree.c/4827
 111 2981 0           Ngblist     63.3625   26866.7523  subfind_density()/subfind_density.c/78
 111 2982 0         Dist2list    126.7250   26993.4773  subfind_density()/subfind_density.c/79
 111 2983 0    DataIndexTable     21.8182   27015.2954  subfind_density()/subfind_density.c/87
  97 2986 0             Right    126.7255   27326.6838  subfind_density()/subfind_density.c/92
  97 2987 0              Todo     15.8407   27342.5245  subfind_density()/subfind_density.c/93
  97 2988 0             DM_Vx    126.7255   27469.2500  subfind_density()/subfind_density.c/95
  97 2989 0             DM_Vy    126.7255   27595.9755  subfind_density()/subfind_density.c/96
  97 2990 0             DM_Vz    126.7255   27722.7010  subfind_density()/subfind_density.c/97
  97 2991 0    	  DensDataGet    123.1544   27845.8553  subfind_density()/subfind_density.c/161
  30 2987 0              Todo     15.8485   27344.2632  subfind_density()/subfind_density.c/93
  30 2988 0             DM_Vx    126.7878   27471.0511  subfind_density()/subfind_density.c/95
  30 2989 0             DM_Vy    126.7878   27597.8389  subfind_density()/subfind_density.c/96
  16 2976 0   DomainNodeIndex      0.3498   19507.7107  force_treeallocate()/forcetree.c/4800
  16 2977 0        Nodes_base   3737.6087   23245.3195  force_treeallocate()/forcetree.c/4803
 111 2984 0      DataNodeList     58.1818   27073.4772  subfind_density()/subfind_density.c/89
 111 2985 0              Left    126.7250   27200.2022  subfind_density()/subfind_density.c/91
 111 2986 0             Right    126.7250   27326.9272  subfind_density()/subfind_density.c/92
 111 2987 0              Todo     15.8406   27342.7679  subfind_density()/subfind_density.c/93
  96 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  96 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  30 2990 0             DM_Vz    126.7878   27724.6268  subfind_density()/subfind_density.c/97
  30 2991 0    	  DensDataGet    125.8297   27850.4564  subfind_density()/subfind_density.c/161
  30 2992 0  	  DensDataResult     94.3722   27944.8286  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  16 2978 0     Extnodes_base   3114.6739   26359.9934  force_treeallocate()/forcetree.c/4811
  16 2979 0          Nextnode    222.3054   26582.2987  force_treeallocate()/forcetree.c/4820
  16 2980 0            Father    221.9056   26804.2043  force_treeallocate()/forcetree.c/4827
 111 2988 0             DM_Vx    126.7250   27469.4929  subfind_density()/subfind_density.c/95
 111 2989 0             DM_Vy    126.7250   27596.2178  subfind_density()/subfind_density.c/96
 111 2990 0             DM_Vz    126.7250   27722.9428  subfind_density()/subfind_density.c/97
  97 2992 0  	  DensDataResult     92.3658   27938.2211  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=97: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=61.7789 MB).
task 97: endrun called with an error level of 812



Task=30: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.1714 MB).
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  16 2981 0           Ngblist     63.4301   26867.6344  subfind_density()/subfind_density.c/78
  16 2982 0         Dist2list    126.8602   26994.4946  subfind_density()/subfind_density.c/79
 111 2991 0    	  DensDataGet    116.6693   27839.6122  subfind_density()/subfind_density.c/161
 111 2992 0  	  DensDataResult     87.5020   27927.1141  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  96 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  96 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  96 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
task 30: endrun called with an error level of 812


   3 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   3 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   3 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   3 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   3 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   3 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   3 2973 0         FOF_PList    380.4234   18861.2799  fof_fof()/fof.c/249
   3 2974 0         FOF_GList    634.0390   19495.3189  fof_fof()/fof.c/337
   3 2975 0             Group     12.5414   19507.8604  fof_fof()/fof.c/380
   3 2976 0   DomainNodeIndex      0.3498   19508.2102  force_treeallocate()/forcetree.c/4800
   3 2977 0        Nodes_base   3737.6087   23245.8189  force_treeallocate()/forcetree.c/4803
   3 2978 0     Extnodes_base   3114.6739   26360.4928  force_treeallocate()/forcetree.c/4811
   3 2979 0          Nextnode    222.3054   26582.7982  force_treeallocate()/forcetree.c/4820
   3 2980 0            Father    221.9056   26804.7038  force_treeallocate()/forcetree.c/4827
  16 2983 0    DataIndexTable     21.8182   27016.3127  subfind_density()/subfind_density.c/87
  16 2984 0      DataNodeList     58.1818   27074.4945  subfind_density()/subfind_density.c/89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=111: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=72.8859 MB).
task 111: endrun called with an error level of 812


 124 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 124 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 124 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 124 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 124 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 124 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 124 2973 0         FOF_PList    380.5393   18861.3958  fof_fof()/fof.c/249
 124 2974 0         FOF_GList    634.2321   19495.6280  fof_fof()/fof.c/337
 124 2975 0             Group     12.0652   19507.6931  fof_fof()/fof.c/380
  96 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 30
   3 2981 0           Ngblist     63.4039   26868.1077  subfind_density()/subfind_density.c/78
   3 2982 0         Dist2list    126.8078   26994.9155  subfind_density()/subfind_density.c/79
   3 2983 0    DataIndexTable     21.8182   27016.7337  subfind_density()/subfind_density.c/87
   3 2984 0      DataNodeList     58.1818   27074.9155  subfind_density()/subfind_density.c/89
  16 2985 0              Left    126.8602   27201.3547  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 112 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 112 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 112 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 112 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 112 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 112 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 111
 124 2976 0   DomainNodeIndex      0.3498   19508.0430  force_treeallocate()/forcetree.c/4800
 124 2977 0        Nodes_base   3737.6087   23245.6517  force_treeallocate()/forcetree.c/4803
  96 2973 0         FOF_PList    380.4619   18861.3184  fof_fof()/fof.c/249
  96 2974 0         FOF_GList    634.1031   19495.4215  fof_fof()/fof.c/337
  96 2975 0             Group     13.3238   19508.7453  fof_fof()/fof.c/380
  96 2976 0   DomainNodeIndex      0.3498   19509.0951  force_treeallocate()/forcetree.c/4800
  96 2977 0        Nodes_base   3737.6087   23246.7038  force_treeallocate()/forcetree.c/4803
  96 2978 0     Extnodes_base   3114.6739   26361.3778  force_treeallocate()/forcetree.c/4811
  96 2979 0          Nextnode    222.3054   26583.6831  force_treeallocate()/forcetree.c/4820
  96 2980 0            Father    221.9056   26805.5887  force_treeallocate()/forcetree.c/4827
  96 2981 0           Ngblist     63.4103   26868.9990  subfind_density()/subfind_density.c/78
  96 2982 0         Dist2list    126.8206   26995.8197  subfind_density()/subfind_density.c/79
  96 2983 0    DataIndexTable     21.8182   27017.6378  subfind_density()/subfind_density.c/87
  96 2984 0      DataNodeList     58.1818   27075.8196  subfind_density()/subfind_density.c/89
   3 2985 0              Left    126.8078   27201.7233  subfind_density()/subfind_density.c/91
   3 2986 0             Right    126.8078   27328.5311  subfind_density()/subfind_density.c/92
   3 2987 0              Todo     15.8510   27344.3820  subfind_density()/subfind_density.c/93
   3 2988 0             DM_Vx    126.8078   27471.1898  subfind_density()/subfind_density.c/95
  16 2986 0             Right    126.8602   27328.2148  subfind_density()/subfind_density.c/92
  15 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  15 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  15 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  15 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  15 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  15 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  15 2973 0         FOF_PList    380.3549   18861.2114  fof_fof()/fof.c/249
  15 2974 0         FOF_GList    633.9248   19495.1362  fof_fof()/fof.c/337
  15 2975 0             Group     12.7178   19507.8540  fof_fof()/fof.c/380
  15 2976 0   DomainNodeIndex      0.3498   19508.2038  force_treeallocate()/forcetree.c/4800
  15 2977 0        Nodes_base   3737.6087   23245.8126  force_treeallocate()/forcetree.c/4803
  15 2978 0     Extnodes_base   3114.6739   26360.4865  force_treeallocate()/forcetree.c/4811
  15 2979 0          Nextnode    222.3054   26582.7918  force_treeallocate()/forcetree.c/4820
 112 2973 0         FOF_PList    380.4143   18861.2708  fof_fof()/fof.c/249
 112 2974 0         FOF_GList    634.0238   19495.2946  fof_fof()/fof.c/337
 112 2975 0             Group     12.2136   19507.5082  fof_fof()/fof.c/380
 124 2978 0     Extnodes_base   3114.6739   26360.3256  force_treeallocate()/forcetree.c/4811
 124 2979 0          Nextnode    222.3054   26582.6310  force_treeallocate()/forcetree.c/4820
 124 2980 0            Father    221.9056   26804.5365  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 97
   3 2989 0             DM_Vy    126.8078   27597.9976  subfind_density()/subfind_density.c/96
   3 2990 0             DM_Vz    126.8078   27724.8054  subfind_density()/subfind_density.c/97
   3 2991 0    	  DensDataGet    138.3654   27863.1708  subfind_density()/subfind_density.c/161
  16 2987 0              Todo     15.8575   27344.0724  subfind_density()/subfind_density.c/93
  15 2980 0            Father    221.9056   26804.6974  force_treeallocate()/forcetree.c/4827
  15 2981 0           Ngblist     63.3925   26868.0899  subfind_density()/subfind_density.c/78
  15 2982 0         Dist2list    126.7850   26994.8749  subfind_density()/subfind_density.c/79
  15 2983 0    DataIndexTable     21.8182   27016.6930  subfind_density()/subfind_density.c/87
  15 2984 0      DataNodeList     58.1818   27074.8748  subfind_density()/subfind_density.c/89
 112 2976 0   DomainNodeIndex      0.3498   19507.8580  force_treeallocate()/forcetree.c/4800
 112 2977 0        Nodes_base   3737.6087   23245.4667  force_treeallocate()/forcetree.c/4803
 124 2981 0           Ngblist     63.4232   26867.9598  subfind_density()/subfind_density.c/78
 124 2982 0         Dist2list    126.8464   26994.8062  subfind_density()/subfind_density.c/79
 124 2983 0    DataIndexTable     21.8182   27016.6244  subfind_density()/subfind_density.c/87
  96 2985 0              Left    126.8206   27202.6403  subfind_density()/subfind_density.c/91
  96 2986 0             Right    126.8206   27329.4609  subfind_density()/subfind_density.c/92
  96 2987 0              Todo     15.8526   27345.3135  subfind_density()/subfind_density.c/93
  96 2988 0             DM_Vx    126.8206   27472.1341  subfind_density()/subfind_density.c/95
   3 2992 0  	  DensDataResult    103.7740   27966.9448  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=3: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=33.0552 MB).
task 3: endrun called with an error level of 812


  16 2988 0             DM_Vx    126.8602   27470.9325  subfind_density()/subfind_density.c/95
  15 2985 0              Left    126.7850   27201.6598  subfind_density()/subfind_density.c/91
  15 2986 0             Right    126.7850   27328.4448  subfind_density()/subfind_density.c/92
  15 2987 0              Todo     15.8481   27344.2929  subfind_density()/subfind_density.c/93
 112 2978 0     Extnodes_base   3114.6739   26360.1406  force_treeallocate()/forcetree.c/4811
 112 2979 0          Nextnode    222.3054   26582.4460  force_treeallocate()/forcetree.c/4820
 112 2980 0            Father    221.9056   26804.3516  force_treeallocate()/forcetree.c/4827
 112 2981 0           Ngblist     63.4024   26867.7540  subfind_density()/subfind_density.c/78
 124 2984 0      DataNodeList     58.1818   27074.8062  subfind_density()/subfind_density.c/89
 124 2985 0              Left    126.8464   27201.6526  subfind_density()/subfind_density.c/91
 124 2986 0             Right    126.8464   27328.4990  subfind_density()/subfind_density.c/92
  96 2989 0             DM_Vy    126.8206   27598.9547  subfind_density()/subfind_density.c/96
  96 2990 0             DM_Vz    126.8206   27725.7754  subfind_density()/subfind_density.c/97
  96 2991 0    	  DensDataGet    130.4844   27856.2597  subfind_density()/subfind_density.c/161
  96 2992 0  	  DensDataResult     97.8633   27954.1230  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 3
  16 2989 0             DM_Vy    126.8602   27597.7927  subfind_density()/subfind_density.c/96
  15 2988 0             DM_Vx    126.7850   27471.0778  subfind_density()/subfind_density.c/95
  15 2989 0             DM_Vy    126.7850   27597.8628  subfind_density()/subfind_density.c/96
  15 2990 0             DM_Vz    126.7850   27724.6478  subfind_density()/subfind_density.c/97
 112 2982 0         Dist2list    126.8048   26994.5588  subfind_density()/subfind_density.c/79
 112 2983 0    DataIndexTable     21.8182   27016.3769  subfind_density()/subfind_density.c/87
 124 2987 0              Todo     15.8558   27344.3548  subfind_density()/subfind_density.c/93

Task=96: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.877 MB).
task 96: endrun called with an error level of 812


  16 2990 0             DM_Vz    126.8602   27724.6528  subfind_density()/subfind_density.c/97
  16 2991 0    	  DensDataGet    114.2828   27838.9356  subfind_density()/subfind_density.c/161
  15 2991 0    	  DensDataGet    117.3122   27841.9599  subfind_density()/subfind_density.c/161
  15 2992 0  	  DensDataResult     87.9841   27929.9441  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 112 2984 0      DataNodeList     58.1818   27074.5587  subfind_density()/subfind_density.c/89
 112 2985 0              Left    126.8048   27201.3635  subfind_density()/subfind_density.c/91
 112 2986 0             Right    126.8048   27328.1683  subfind_density()/subfind_density.c/92
 124 2988 0             DM_Vx    126.8464   27471.2012  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 96
  16 2992 0  	  DensDataResult     85.7121   27924.6477  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=15: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.0559 MB).
task 15: endrun called with an error level of 812


 112 2987 0              Todo     15.8506   27344.0189  subfind_density()/subfind_density.c/93
 124 2989 0             DM_Vy    126.8464   27598.0477  subfind_density()/subfind_density.c/96
 124 2990 0             DM_Vz    126.8464   27724.8941  subfind_density()/subfind_density.c/97
 124 2991 0    	  DensDataGet    111.7656   27836.6597  subfind_density()/subfind_density.c/161
 An HDF5 error was detected, we'd better stop.

Task=16: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.3523 MB).
task 16: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 15
 112 2988 0             DM_Vx    126.8048   27470.8236  subfind_density()/subfind_density.c/95
 124 2992 0  	  DensDataResult     83.8242   27920.4838  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
HDF5-DIAG: Error detected in HDF5 (1.8.12) application called MPI_Abort(MPI_COMM_WORLD, 812) - process 16
 112 2989 0             DM_Vy    126.8048   27597.6284  subfind_density()/subfind_density.c/96

Task=124: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.5162 MB).
task 124: endrun called with an error level of 812


MPI-process 96:
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 An HDF5 error was detected, we'd better stop.
 112 2990 0             DM_Vz    126.8048   27724.4331  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 124
task 96: endrun called with an error level of 124


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  87 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  41 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  41 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  41 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  41 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  41 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  41 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  41 2973 0         FOF_PList    380.5523   18861.4088  fof_fof()/fof.c/249
  41 2974 0         FOF_GList    634.2538   19495.6627  fof_fof()/fof.c/337
  41 2975 0             Group     11.3119   19506.9746  fof_fof()/fof.c/380
  41 2976 0   DomainNodeIndex      0.3498   19507.3244  force_treeallocate()/forcetree.c/4800
  41 2977 0        Nodes_base   3737.6087   23244.9331  force_treeallocate()/forcetree.c/4803
  41 2978 0     Extnodes_base   3114.6739   26359.6070  force_treeallocate()/forcetree.c/4811
  41 2979 0          Nextnode    222.3054   26581.9124  force_treeallocate()/forcetree.c/4820
  41 2980 0            Father    221.9056   26803.8180  force_treeallocate()/forcetree.c/4827
task 16: endrun called with an error level of 124


 112 2991 0    	  DensDataGet    126.4161   27850.8492  subfind_density()/subfind_density.c/161
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 96
  86 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  86 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  86 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  86 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  86 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  86 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  86 2973 0         FOF_PList    380.6286   18861.4852  fof_fof()/fof.c/249
  86 2974 0         FOF_GList    634.3810   19495.8662  fof_fof()/fof.c/337
  86 2975 0             Group     11.1978   19507.0639  fof_fof()/fof.c/380
  86 2976 0   DomainNodeIndex      0.3498   19507.4137  force_treeallocate()/forcetree.c/4800
  86 2977 0        Nodes_base   3737.6087   23245.0225  force_treeallocate()/forcetree.c/4803
  86 2978 0     Extnodes_base   3114.6739   26359.6964  force_treeallocate()/forcetree.c/4811
  86 2979 0          Nextnode    222.3054   26582.0017  force_treeallocate()/forcetree.c/4820
  40 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  40 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  40 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  40 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  40 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  40 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  40 2973 0         FOF_PList    380.5726   18861.4291  fof_fof()/fof.c/249
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 16:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 16
 112 2992 0  	  DensDataResult     94.8121   27945.6613  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  87 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  87 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  87 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  87 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  87 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  87 2973 0         FOF_PList    380.6225   18861.4790  fof_fof()/fof.c/249
  87 2974 0         FOF_GList    634.3708   19495.8498  fof_fof()/fof.c/337
  87 2975 0             Group     13.5386   19509.3884  fof_fof()/fof.c/380
  87 2976 0   DomainNodeIndex      0.3498   19509.7382  force_treeallocate()/forcetree.c/4800
  87 2977 0        Nodes_base   3737.6087   23247.3469  force_treeallocate()/forcetree.c/4803
  41 2981 0           Ngblist     63.4254   26867.2434  subfind_density()/subfind_density.c/78
  41 2982 0         Dist2list    126.8508   26994.0941  subfind_density()/subfind_density.c/79
  41 2983 0    DataIndexTable     21.8182   27015.9123  subfind_density()/subfind_density.c/87
  41 2984 0      DataNodeList     58.1818   27074.0941  subfind_density()/subfind_density.c/89
  41 2985 0              Left    126.8508   27200.9449  subfind_density()/subfind_density.c/91
  41 2986 0             Right    126.8508   27327.7956  subfind_density()/subfind_density.c/92
  41 2987 0              Todo     15.8563   27343.6520  subfind_density()/subfind_density.c/93
  41 2988 0             DM_Vx    126.8508   27470.5028  subfind_density()/subfind_density.c/95
  41 2989 0             DM_Vy    126.8508   27597.3535  subfind_density()/subfind_density.c/96
  41 2990 0             DM_Vz    126.8508   27724.2043  subfind_density()/subfind_density.c/97

Task=112: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=54.3387 MB).
  86 2980 0            Father    221.9056   26803.9073  force_treeallocate()/forcetree.c/4827
  86 2981 0           Ngblist     63.4381   26867.3454  subfind_density()/subfind_density.c/78
  86 2982 0         Dist2list    126.8762   26994.2216  subfind_density()/subfind_density.c/79
  86 2983 0    DataIndexTable     21.8182   27016.0398  subfind_density()/subfind_density.c/87
  86 2984 0      DataNodeList     58.1818   27074.2216  subfind_density()/subfind_density.c/89
  86 2985 0              Left    126.8762   27201.0978  subfind_density()/subfind_density.c/91
  86 2986 0             Right    126.8762   27327.9740  subfind_density()/subfind_density.c/92
  86 2987 0              Todo     15.8595   27343.8335  subfind_density()/subfind_density.c/93
  86 2988 0             DM_Vx    126.8762   27470.7098  subfind_density()/subfind_density.c/95
  40 2974 0         FOF_GList    634.2877   19495.7168  fof_fof()/fof.c/337
  40 2975 0             Group     12.1067   19507.8235  fof_fof()/fof.c/380
  40 2976 0   DomainNodeIndex      0.3498   19508.1733  force_treeallocate()/forcetree.c/4800
  40 2977 0        Nodes_base   3737.6087   23245.7820  force_treeallocate()/forcetree.c/4803
  40 2978 0     Extnodes_base   3114.6739   26360.4560  force_treeallocate()/forcetree.c/4811
task 112: endrun called with an error level of 812


  87 2978 0     Extnodes_base   3114.6739   26362.0208  force_treeallocate()/forcetree.c/4811
  87 2979 0          Nextnode    222.3054   26584.3262  force_treeallocate()/forcetree.c/4820
  87 2980 0            Father    221.9056   26806.2318  force_treeallocate()/forcetree.c/4827
  87 2981 0           Ngblist     63.4371   26869.6689  subfind_density()/subfind_density.c/78
  87 2982 0         Dist2list    126.8742   26996.5430  subfind_density()/subfind_density.c/79
  87 2983 0    DataIndexTable     21.8182   27018.3612  subfind_density()/subfind_density.c/87
  87 2984 0      DataNodeList     58.1818   27076.5430  subfind_density()/subfind_density.c/89
  41 2991 0    	  DensDataGet    114.6934   27838.8977  subfind_density()/subfind_density.c/161
  41 2992 0  	  DensDataResult     86.0200   27924.9177  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=41: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.0823 MB).
task 41: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 112
  86 2989 0             DM_Vy    126.8762   27597.5860  subfind_density()/subfind_density.c/96
  86 2990 0             DM_Vz    126.8762   27724.4622  subfind_density()/subfind_density.c/97
  86 2991 0    	  DensDataGet    133.2980   27857.7601  subfind_density()/subfind_density.c/161
  86 2992 0  	  DensDataResult     99.9735   27957.7336  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  40 2979 0          Nextnode    222.3054   26582.7613  force_treeallocate()/forcetree.c/4820
  40 2980 0            Father    221.9056   26804.6669  force_treeallocate()/forcetree.c/4827
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12)   87 2985 0              Left    126.8742   27203.4172  subfind_density()/subfind_density.c/91
  87 2986 0             Right    126.8742   27330.2913  subfind_density()/subfind_density.c/92
  40 2981 0           Ngblist     63.4288   26868.0957  subfind_density()/subfind_density.c/78
MPI-process 112:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API

Task=86: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=42.2664 MB).
task 86: endrun called with an error level of 812


  40 2982 0         Dist2list    126.8575   26994.9532  subfind_density()/subfind_density.c/79
task 112: endrun called with an error level of 124


  87 2987 0              Todo     15.8593   27346.1506  subfind_density()/subfind_density.c/93
  87 2988 0             DM_Vx    126.8742   27473.0248  subfind_density()/subfind_density.c/95
  40 2983 0    DataIndexTable     21.8182   27016.7714  subfind_density()/subfind_density.c/87
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 112
  87 2989 0             DM_Vy    126.8742   27599.8989  subfind_density()/subfind_density.c/96
  40 2984 0      DataNodeList     58.1818   27074.9532  subfind_density()/subfind_density.c/89
  87 2990 0             DM_Vz    126.8742   27726.7731  subfind_density()/subfind_density.c/97
  40 2985 0              Left    126.8575   27201.8107  subfind_density()/subfind_density.c/91
  87 2991 0    	  DensDataGet    111.0345   27837.8076  subfind_density()/subfind_density.c/161
  40 2986 0             Right    126.8575   27328.6683  subfind_density()/subfind_density.c/92
  87 2992 0  	  DensDataResult     83.2759   27921.0835  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  40 2987 0              Todo     15.8572   27344.5255  subfind_density()/subfind_density.c/93

Task=87: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.9165 MB).
task 87: endrun called with an error level of 812


  40 2988 0             DM_Vx    126.8575   27471.3830  subfind_density()/subfind_density.c/95
  40 2989 0             DM_Vy    126.8575   27598.2405  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 86
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 41
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 87
  40 2990 0             DM_Vz    126.8575   27725.0981  subfind_density()/subfind_density.c/97
  40 2991 0    	  DensDataGet    136.0037   27861.1017  subfind_density()/subfind_density.c/161
  40 2992 0  	  DensDataResult    102.0027   27963.1045  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=40: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=36.8955 MB).
task 40: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 40
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 40:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 40: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 40
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   6 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   6 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   6 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   6 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   6 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   6 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   6 2973 0         FOF_PList    380.4730   18861.3295  fof_fof()/fof.c/249
   6 2974 0         FOF_GList    634.1216   19495.4511  fof_fof()/fof.c/337
   6 2975 0             Group     10.9197   19506.3708  fof_fof()/fof.c/380
 121 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 121 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 121 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 121 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 121 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 121 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 121 2973 0         FOF_PList    380.5840   18861.4405  fof_fof()/fof.c/249
 121 2974 0         FOF_GList    634.3067   19495.7472  fof_fof()/fof.c/337
 121 2975 0             Group     11.0255   19506.7727  fof_fof()/fof.c/380
 121 2976 0   DomainNodeIndex      0.3498   19507.1226  force_treeallocate()/forcetree.c/4800
 121 2977 0        Nodes_base   3737.6087   23244.7313  force_treeallocate()/forcetree.c/4803
 121 2978 0     Extnodes_base   3114.6739   26359.4052  force_treeallocate()/forcetree.c/4811
 121 2979 0          Nextnode    222.3054   26581.7105  force_treeallocate()/forcetree.c/4820
 121 2980 0            Father    221.9056   26803.6161  force_treeallocate()/forcetree.c/4827
   6 2976 0   DomainNodeIndex      0.3498   19506.7206  force_treeallocate()/forcetree.c/4800
   6 2977 0        Nodes_base   3737.6087   23244.3293  force_treeallocate()/forcetree.c/4803
 121 2981 0           Ngblist     63.4307   26867.0468  subfind_density()/subfind_density.c/78
 121 2982 0         Dist2list    126.8613   26993.9081  subfind_density()/subfind_density.c/79
 121 2983 0    DataIndexTable     21.8182   27015.7263  subfind_density()/subfind_density.c/87
 121 2984 0      DataNodeList     58.1818   27073.9081  subfind_density()/subfind_density.c/89
   6 2978 0     Extnodes_base   3114.6739   26359.0033  force_treeallocate()/forcetree.c/4811
   6 2979 0          Nextnode    222.3054   26581.3086  force_treeallocate()/forcetree.c/4820
   6 2980 0            Father    221.9056   26803.2142  force_treeallocate()/forcetree.c/4827
 121 2985 0              Left    126.8613   27200.7694  subfind_density()/subfind_density.c/91
 121 2986 0             Right    126.8613   27327.6308  subfind_density()/subfind_density.c/92
 121 2987 0              Todo     15.8577   27343.4885  subfind_density()/subfind_density.c/93
   6 2981 0           Ngblist     63.4122   26866.6264  subfind_density()/subfind_density.c/78
   6 2982 0         Dist2list    126.8243   26993.4507  subfind_density()/subfind_density.c/79
   6 2983 0    DataIndexTable     21.8182   27015.2689  subfind_density()/subfind_density.c/87
 121 2988 0             DM_Vx    126.8613   27470.3498  subfind_density()/subfind_density.c/95
 121 2989 0             DM_Vy    126.8613   27597.2111  subfind_density()/subfind_density.c/96
 121 2990 0             DM_Vz    126.8613   27724.0725  subfind_density()/subfind_density.c/97
   6 2984 0      DataNodeList     58.1818   27073.4507  subfind_density()/subfind_density.c/89
   6 2985 0              Left    126.8243   27200.2750  subfind_density()/subfind_density.c/91
   6 2986 0             Right    126.8243   27327.0993  subfind_density()/subfind_density.c/92
 121 2991 0    	  DensDataGet    132.6010   27856.6735  subfind_density()/subfind_density.c/161
 121 2992 0  	  DensDataResult     99.4508   27956.1242  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   6 2987 0              Todo     15.8530   27342.9524  subfind_density()/subfind_density.c/93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=121: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=43.8758 MB).
task 121: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   6 2988 0             DM_Vx    126.8243   27469.7767  subfind_density()/subfind_density.c/95
   2 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   2 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   2 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   2 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   2 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   2 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   2 2973 0         FOF_PList    380.5284   18861.3849  fof_fof()/fof.c/249
   2 2974 0         FOF_GList    634.2139   19495.5988  fof_fof()/fof.c/337
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 121
 125 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 125 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 125 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 125 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 125 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 125 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 125 2973 0         FOF_PList    380.6345   18861.4911  fof_fof()/fof.c/249
 125 2974 0         FOF_GList    634.3909   19495.8820  fof_fof()/fof.c/337
 125 2975 0             Group     12.3235   19508.2055  fof_fof()/fof.c/380
 125 2976 0   DomainNodeIndex      0.3498   19508.5554  force_treeallocate()/forcetree.c/4800
 125 2977 0        Nodes_base   3737.6087   23246.1641  force_treeallocate()/forcetree.c/4803
 125 2978 0     Extnodes_base   3114.6739   26360.8380  force_treeallocate()/forcetree.c/4811
   6 2989 0             DM_Vy    126.8243   27596.6010  subfind_density()/subfind_density.c/96
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
   2 2975 0             Group     13.6444   19509.2433  fof_fof()/fof.c/380
   2 2976 0   DomainNodeIndex      0.3498   19509.5931  force_treeallocate()/forcetree.c/4800
   2 2977 0        Nodes_base   3737.6087   23247.2018  force_treeallocate()/forcetree.c/4803
 125 2979 0          Nextnode    222.3054   26583.1433  force_treeallocate()/forcetree.c/4820
 125 2980 0            Father    221.9056   26805.0489  force_treeallocate()/forcetree.c/4827
 125 2981 0           Ngblist     63.4391   26868.4880  subfind_density()/subfind_density.c/78
 125 2982 0         Dist2list    126.8782   26995.3662  subfind_density()/subfind_density.c/79
   6 2990 0             DM_Vz    126.8243   27723.4253  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  85 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  85 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  85 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  85 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  85 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  85 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  85 2973 0         FOF_PList    380.4116   18861.2682  fof_fof()/fof.c/249
  85 2974 0         FOF_GList    634.0194   19495.2876  fof_fof()/fof.c/337
  85 2975 0             Group     10.9197   19506.2072  fof_fof()/fof.c/380
  85 2976 0   DomainNodeIndex      0.3498   19506.5571  force_treeallocate()/forcetree.c/4800
  85 2977 0        Nodes_base   3737.6087   23244.1658  force_treeallocate()/forcetree.c/4803
  85 2978 0     Extnodes_base   3114.6739   26358.8397  force_treeallocate()/forcetree.c/4811
  85 2979 0          Nextnode    222.3054   26581.1451  force_treeallocate()/forcetree.c/4820
  85 2980 0            Father    221.9056   26803.0506  force_treeallocate()/forcetree.c/4827
   2 2978 0     Extnodes_base   3114.6739   26361.8757  force_treeallocate()/forcetree.c/4811
   2 2979 0          Nextnode    222.3054   26584.1811  force_treeallocate()/forcetree.c/4820
 125 2983 0    DataIndexTable     21.8182   27017.1844  subfind_density()/subfind_density.c/87
 125 2984 0      DataNodeList     58.1818   27075.3662  subfind_density()/subfind_density.c/89
 125 2985 0              Left    126.8782   27202.2444  subfind_density()/subfind_density.c/91
   6 2991 0    	  DensDataGet    109.4792   27832.9046  subfind_density()/subfind_density.c/161
  42 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  42 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  42 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  42 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  42 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  42 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  42 2973 0         FOF_PList    380.4749   18861.3314  fof_fof()/fof.c/249
  42 2974 0         FOF_GList    634.1248   19495.4563  fof_fof()/fof.c/337
  42 2975 0             Group     12.8372   19508.2934  fof_fof()/fof.c/380
  42 2976 0   DomainNodeIndex      0.3498   19508.6432  force_treeallocate()/forcetree.c/4800
  85 2981 0           Ngblist     63.4019   26866.4526  subfind_density()/subfind_density.c/78
  85 2982 0         Dist2list    126.8039   26993.2565  subfind_density()/subfind_density.c/79
  85 2983 0    DataIndexTable     21.8182   27015.0746  subfind_density()/subfind_density.c/87
  85 2984 0      DataNodeList     58.1818   27073.2564  subfind_density()/subfind_density.c/89
   2 2980 0            Father    221.9056   26806.0867  force_treeallocate()/forcetree.c/4827
   2 2981 0           Ngblist     63.4214   26869.5080  subfind_density()/subfind_density.c/78
   2 2982 0         Dist2list    126.8428   26996.3508  subfind_density()/subfind_density.c/79
 125 2986 0             Right    126.8782   27329.1225  subfind_density()/subfind_density.c/92
 125 2987 0              Todo     15.8598   27344.9823  subfind_density()/subfind_density.c/93
 125 2988 0             DM_Vx    126.8782   27471.8605  subfind_density()/subfind_density.c/95
 125 2989 0             DM_Vy    126.8782   27598.7387  subfind_density()/subfind_density.c/96
   6 2992 0  	  DensDataResult     82.1094   27915.0140  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  42 2977 0        Nodes_base   3737.6087   23246.2519  force_treeallocate()/forcetree.c/4803
  85 2985 0              Left    126.8039   27200.0603  subfind_density()/subfind_density.c/91
  85 2986 0             Right    126.8039   27326.8642  subfind_density()/subfind_density.c/92
  85 2987 0              Todo     15.8505   27342.7147  subfind_density()/subfind_density.c/93
   2 2983 0    DataIndexTable     21.8182   27018.1690  subfind_density()/subfind_density.c/87
   2 2984 0      DataNodeList     58.1818   27076.3508  subfind_density()/subfind_density.c/89
   2 2985 0              Left    126.8428   27203.1936  subfind_density()/subfind_density.c/91
 125 2990 0             DM_Vz    126.8782   27725.6169  subfind_density()/subfind_density.c/97
 125 2991 0    	  DensDataGet    128.0898   27853.7067  subfind_density()/subfind_density.c/161
 125 2992 0  	  DensDataResult     96.0674   27949.7741  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=6: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.986 MB).
task 6: endrun called with an error level of 812


  42 2978 0     Extnodes_base   3114.6739   26360.9259  force_treeallocate()/forcetree.c/4811
  42 2979 0          Nextnode    222.3054   26583.2312  force_treeallocate()/forcetree.c/4820
  42 2980 0            Father    221.9056   26805.1368  force_treeallocate()/forcetree.c/4827
  85 2988 0             DM_Vx    126.8039   27469.5186  subfind_density()/subfind_density.c/95
  85 2989 0             DM_Vy    126.8039   27596.3224  subfind_density()/subfind_density.c/96
  85 2990 0             DM_Vz    126.8039   27723.1263  subfind_density()/subfind_density.c/97
   2 2986 0             Right    126.8428   27330.0364  subfind_density()/subfind_density.c/92
   2 2987 0              Todo     15.8554   27345.8917  subfind_density()/subfind_density.c/93

Task=125: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=50.2259 MB).
task 125: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 6
  42 2981 0           Ngblist     63.4125   26868.5493  subfind_density()/subfind_density.c/78
  42 2982 0         Dist2list    126.8250   26995.3743  subfind_density()/subfind_density.c/79
  42 2983 0    DataIndexTable     21.8182   27017.1925  subfind_density()/subfind_density.c/87
  85 2991 0    	  DensDataGet    126.3121   27849.4384  subfind_density()/subfind_density.c/161
  85 2992 0  	  DensDataResult     94.7341   27944.1724  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
   2 2988 0             DM_Vx    126.8428   27472.7345  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 125
  42 2984 0      DataNodeList     58.1818   27075.3742  subfind_density()/subfind_density.c/89
  42 2985 0              Left    126.8250   27202.1992  subfind_density()/subfind_density.c/91
  42 2986 0             Right    126.8250   27329.0242  subfind_density()/subfind_density.c/92

Task=85: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.8276 MB).
   2 2989 0             DM_Vy    126.8428   27599.5773  subfind_density()/subfind_density.c/96
  42 2987 0              Todo     15.8531   27344.8773  subfind_density()/subfind_density.c/93
task 85: endrun called with an error level of 812


   2 2990 0             DM_Vz    126.8428   27726.4201  subfind_density()/subfind_density.c/97
  42 2988 0             DM_Vx    126.8250   27471.7023  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 85
   2 2991 0    	  DensDataGet    113.7430   27840.1631  subfind_density()/subfind_density.c/161
  42 2989 0             DM_Vy    126.8250   27598.5272  subfind_density()/subfind_density.c/96
   2 2992 0  	  DensDataResult     85.3072   27925.4703  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------  42 2990 0             DM_Vz    126.8250   27725.3522  subfind_density()/subfind_density.c/97

  42 2991 0    	  DensDataGet    114.7303   27840.0826  subfind_density()/subfind_density.c/161

Task=2: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.5297 MB).
  42 2992 0  	  DensDataResult     86.0478   27926.1303  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------task 2: endrun called with an error level of 812



application called MPI_Abort(MPI_COMM_WORLD, 812) - process 2

Task=42: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.8697 MB).
task 42: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 42
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  91 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  91 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  91 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  91 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  91 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  91 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  91 2973 0         FOF_PList    380.6394   18861.4960  fof_fof()/fof.c/249
  91 2974 0         FOF_GList    634.3990   19495.8950  fof_fof()/fof.c/337
  91 2975 0             Group     11.7404   19507.6354  fof_fof()/fof.c/380
  91 2976 0   DomainNodeIndex      0.3498   19507.9852  force_treeallocate()/forcetree.c/4800
  91 2977 0        Nodes_base   3737.6087   23245.5939  force_treeallocate()/forcetree.c/4803
  91 2978 0     Extnodes_base   3114.6739   26360.2678  force_treeallocate()/forcetree.c/4811
  91 2979 0          Nextnode    222.3054   26582.5732  force_treeallocate()/forcetree.c/4820
  91 2980 0            Father    221.9056   26804.4788  force_treeallocate()/forcetree.c/4827
  91 2981 0           Ngblist     63.4399   26867.9187  subfind_density()/subfind_density.c/78
  91 2982 0         Dist2list    126.8798   26994.7985  subfind_density()/subfind_density.c/79
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  37 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  37 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  37 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  37 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  37 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  37 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  37 2973 0         FOF_PList    380.0064   18860.8629  fof_fof()/fof.c/249
  37 2974 0         FOF_GList    633.3440   19494.2069  fof_fof()/fof.c/337
  37 2975 0             Group     13.9038   19508.1107  fof_fof()/fof.c/380
  37 2976 0   DomainNodeIndex      0.3498   19508.4606  force_treeallocate()/forcetree.c/4800
  37 2977 0        Nodes_base   3737.6087   23246.0693  force_treeallocate()/forcetree.c/4803
  37 2978 0     Extnodes_base   3114.6739   26360.7432  force_treeallocate()/forcetree.c/4811
  37 2979 0          Nextnode    222.3054   26583.0486  force_treeallocate()/forcetree.c/4820
  37 2980 0            Father    221.9056   26804.9541  force_treeallocate()/forcetree.c/4827
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  91 2983 0    DataIndexTable     21.8182   27016.6167  subfind_density()/subfind_density.c/87
  91 2984 0      DataNodeList     58.1818   27074.7985  subfind_density()/subfind_density.c/89
  91 2985 0              Left    126.8798   27201.6783  subfind_density()/subfind_density.c/91
  91 2986 0             Right    126.8798   27328.5581  subfind_density()/subfind_density.c/92
  91 2987 0              Todo     15.8600   27344.4181  subfind_density()/subfind_density.c/93
  43 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  43 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  43 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  43 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  43 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  43 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  43 2973 0         FOF_PList    380.3750   18861.2316  fof_fof()/fof.c/249
  43 2974 0         FOF_GList    633.9584   19495.1900  fof_fof()/fof.c/337
  43 2975 0             Group     10.9197   19506.1096  fof_fof()/fof.c/380
  43 2976 0   DomainNodeIndex      0.3498   19506.4595  force_treeallocate()/forcetree.c/4800
 108 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 108 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 108 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 108 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 108 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 108 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 108 2973 0         FOF_PList    380.5143   18861.3708  fof_fof()/fof.c/249
 108 2974 0         FOF_GList    634.1905   19495.5614  fof_fof()/fof.c/337
 108 2975 0             Group     11.7622   19507.3236  fof_fof()/fof.c/380
 108 2976 0   DomainNodeIndex      0.3498   19507.6734  force_treeallocate()/forcetree.c/4800
 108 2977 0        Nodes_base   3737.6087   23245.2821  force_treeallocate()/forcetree.c/4803
 108 2978 0     Extnodes_base   3114.6739   26359.9560  force_treeallocate()/forcetree.c/4811
 108 2979 0          Nextnode    222.3054   26582.2614  force_treeallocate()/forcetree.c/4820
  37 2981 0           Ngblist     63.3344   26868.2886  subfind_density()/subfind_density.c/78
  37 2982 0         Dist2list    126.6688   26994.9574  subfind_density()/subfind_density.c/79
  37 2983 0    DataIndexTable     21.8182   27016.7755  subfind_density()/subfind_density.c/87
  37 2984 0      DataNodeList     58.1818   27074.9573  subfind_density()/subfind_density.c/89
  19 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  19 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  19 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  19 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  19 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  19 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  19 2973 0         FOF_PList    380.6385   18861.4950  fof_fof()/fof.c/249
  19 2974 0         FOF_GList    634.3975   19495.8925  fof_fof()/fof.c/337
  19 2975 0             Group     13.3809   19509.2734  fof_fof()/fof.c/380
  19 2976 0   DomainNodeIndex      0.3498   19509.6232  force_treeallocate()/forcetree.c/4800
  19 2977 0        Nodes_base   3737.6087   23247.2319  force_treeallocate()/forcetree.c/4803
  19 2978 0     Extnodes_base   3114.6739   26361.9058  force_treeallocate()/forcetree.c/4811
  19 2979 0          Nextnode    222.3054   26584.2112  force_treeallocate()/forcetree.c/4820
  19 2980 0            Father    221.9056   26806.1168  force_treeallocate()/forcetree.c/4827
  19 2981 0           Ngblist     63.4398   26869.5566  subfind_density()/subfind_density.c/78
  19 2982 0         Dist2list    126.8795   26996.4361  subfind_density()/subfind_density.c/79
  19 2983 0    DataIndexTable     21.8182   27018.2542  subfind_density()/subfind_density.c/87
  91 2988 0             DM_Vx    126.8798   27471.2979  subfind_density()/subfind_density.c/95
  91 2989 0             DM_Vy    126.8798   27598.1777  subfind_density()/subfind_density.c/96
  91 2990 0             DM_Vz    126.8798   27725.0575  subfind_density()/subfind_density.c/97
  43 2977 0        Nodes_base   3737.6087   23244.0682  force_treeallocate()/forcetree.c/4803
  43 2978 0     Extnodes_base   3114.6739   26358.7421  force_treeallocate()/forcetree.c/4811
  43 2979 0          Nextnode    222.3054   26581.0475  force_treeallocate()/forcetree.c/4820
 108 2980 0            Father    221.9056   26804.1670  force_treeallocate()/forcetree.c/4827
 108 2981 0           Ngblist     63.4191   26867.5860  subfind_density()/subfind_density.c/78
  37 2985 0              Left    126.6688   27201.6261  subfind_density()/subfind_density.c/91
  37 2986 0             Right    126.6688   27328.2949  subfind_density()/subfind_density.c/92
  37 2987 0              Todo     15.8336   27344.1285  subfind_density()/subfind_density.c/93
  19 2984 0      DataNodeList     58.1818   27076.4360  subfind_density()/subfind_density.c/89
  19 2985 0              Left    126.8795   27203.3155  subfind_density()/subfind_density.c/91
  19 2986 0             Right    126.8795   27330.1950  subfind_density()/subfind_density.c/92
  19 2987 0              Todo     15.8599   27346.0550  subfind_density()/subfind_density.c/93
  19 2988 0             DM_Vx    126.8795   27472.9345  subfind_density()/subfind_density.c/95
  19 2989 0             DM_Vy    126.8795   27599.8140  subfind_density()/subfind_density.c/96
  91 2991 0    	  DensDataGet    116.7166   27841.7741  subfind_density()/subfind_density.c/161
  91 2992 0  	  DensDataResult     87.5375   27929.3116  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=91: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.6884 MB).
  43 2980 0            Father    221.9056   26802.9530  force_treeallocate()/forcetree.c/4827
  43 2981 0           Ngblist     63.3958   26866.3489  subfind_density()/subfind_density.c/78
  43 2982 0         Dist2list    126.7917   26993.1406  subfind_density()/subfind_density.c/79
 108 2982 0         Dist2list    126.8381   26994.4241  subfind_density()/subfind_density.c/79
 108 2983 0    DataIndexTable     21.8182   27016.2423  subfind_density()/subfind_density.c/87
 108 2984 0      DataNodeList     58.1818   27074.4241  subfind_density()/subfind_density.c/89
  37 2988 0             DM_Vx    126.6688   27470.7973  subfind_density()/subfind_density.c/95
  37 2989 0             DM_Vy    126.6688   27597.4661  subfind_density()/subfind_density.c/96
  37 2990 0             DM_Vz    126.6688   27724.1349  subfind_density()/subfind_density.c/97
  37 2991 0    	  DensDataGet    123.6744   27847.8094  subfind_density()/subfind_density.c/161
  19 2990 0             DM_Vz    126.8795   27726.6935  subfind_density()/subfind_density.c/97
  19 2991 0    	  DensDataGet    138.9404   27865.6339  subfind_density()/subfind_density.c/161
task 91: endrun called with an error level of 812


  43 2983 0    DataIndexTable     21.8182   27014.9587  subfind_density()/subfind_density.c/87
  43 2984 0      DataNodeList     58.1818   27073.1405  subfind_density()/subfind_density.c/89
  43 2985 0              Left    126.7917   27199.9322  subfind_density()/subfind_density.c/91
 108 2985 0              Left    126.8381   27201.2622  subfind_density()/subfind_density.c/91
 108 2986 0             Right    126.8381   27328.1003  subfind_density()/subfind_density.c/92
 108 2987 0              Todo     15.8548   27343.9551  subfind_density()/subfind_density.c/93
  37 2992 0  	  DensDataResult     92.7558   27940.5652  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=37: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.4348 MB).
  19 2992 0  	  DensDataResult    104.2053   27969.8392  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=19: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=30.1608 MB).
task 19: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 91
  43 2986 0             Right    126.7917   27326.7239  subfind_density()/subfind_density.c/92
  43 2987 0              Todo     15.8490   27342.5729  subfind_density()/subfind_density.c/93
  43 2988 0             DM_Vx    126.7917   27469.3645  subfind_density()/subfind_density.c/95
 108 2988 0             DM_Vx    126.8381   27470.7932  subfind_density()/subfind_density.c/95
 108 2989 0             DM_Vy    126.8381   27597.6313  subfind_density()/subfind_density.c/96
task 37: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 19
 116 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 116 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 116 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 116 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 116 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 116 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 116 2973 0         FOF_PList    380.0620   18860.9186  fof_fof()/fof.c/249
 116 2974 0         FOF_GList    633.4367   19494.3553  fof_fof()/fof.c/337
 116 2975 0             Group     10.9197   19505.2750  fof_fof()/fof.c/380
  43 2989 0             DM_Vy    126.7917   27596.1562  subfind_density()/subfind_density.c/96
 108 2990 0             DM_Vz    126.8381   27724.4694  subfind_density()/subfind_density.c/97
 108 2991 0    	  DensDataGet    144.4670   27868.9364  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 37
 116 2976 0   DomainNodeIndex      0.3498   19505.6248  force_treeallocate()/forcetree.c/4800
 116 2977 0        Nodes_base   3737.6087   23243.2335  force_treeallocate()/forcetree.c/4803
 116 2978 0     Extnodes_base   3114.6739   26357.9074  force_treeallocate()/forcetree.c/4811
  43 2990 0             DM_Vz    126.7917   27722.9479  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 108 2992 0  	  DensDataResult    108.3502   27977.2866  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=108: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=22.7134 MB).
task 108: endrun called with an error level of 812


 116 2979 0          Nextnode    222.3054   26580.2128  force_treeallocate()/forcetree.c/4820
 116 2980 0            Father    221.9056   26802.1184  force_treeallocate()/forcetree.c/4827
 116 2981 0           Ngblist     63.3437   26865.4621  subfind_density()/subfind_density.c/78
  43 2991 0    	  DensDataGet    120.7590   27843.7069  subfind_density()/subfind_density.c/161
 106 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 106 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 106 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 106 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 106 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 106 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 106 2973 0         FOF_PList    380.6106   18861.4671  fof_fof()/fof.c/249
 106 2974 0         FOF_GList    634.3510   19495.8181  fof_fof()/fof.c/337
 106 2975 0             Group     12.8278   19508.6460  fof_fof()/fof.c/380
 106 2976 0   DomainNodeIndex      0.3498   19508.9958  force_treeallocate()/forcetree.c/4800
 106 2977 0        Nodes_base   3737.6087   23246.6045  force_treeallocate()/forcetree.c/4803
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 108
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 116 2982 0         Dist2list    126.6873   26992.1494  subfind_density()/subfind_density.c/79
 116 2983 0    DataIndexTable     21.8182   27013.9676  subfind_density()/subfind_density.c/87
 116 2984 0      DataNodeList     58.1818   27072.1494  subfind_density()/subfind_density.c/89
  43 2992 0  	  DensDataResult     90.5692   27934.2761  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 106 2978 0     Extnodes_base   3114.6739   26361.2784  force_treeallocate()/forcetree.c/4811
 106 2979 0          Nextnode    222.3054   26583.5838  force_treeallocate()/forcetree.c/4820
  21 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  21 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  21 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  21 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  21 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  21 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  21 2973 0         FOF_PList    380.5552   18861.4118  fof_fof()/fof.c/249
  21 2974 0         FOF_GList    634.2587   19495.6704  fof_fof()/fof.c/337
  21 2975 0             Group     12.9357   19508.6062  fof_fof()/fof.c/380
  21 2976 0   DomainNodeIndex      0.3498   19508.9560  force_treeallocate()/forcetree.c/4800
  21 2977 0        Nodes_base   3737.6087   23246.5647  force_treeallocate()/forcetree.c/4803
  21 2978 0     Extnodes_base   3114.6739   26361.2386  force_treeallocate()/forcetree.c/4811
  21 2979 0          Nextnode    222.3054   26583.5440  force_treeallocate()/forcetree.c/4820
  21 2980 0            Father    221.9056   26805.4496  force_treeallocate()/forcetree.c/4827
  21 2981 0           Ngblist     63.4259   26868.8754  subfind_density()/subfind_density.c/78
  21 2982 0         Dist2list    126.8517   26995.7272  subfind_density()/subfind_density.c/79
  21 2983 0    DataIndexTable     21.8182   27017.5454  subfind_density()/subfind_density.c/87
 116 2985 0              Left    126.6873   27198.8367  subfind_density()/subfind_density.c/91
 116 2986 0             Right    126.6873   27325.5241  subfind_density()/subfind_density.c/92
 116 2987 0              Todo     15.8359   27341.3600  subfind_density()/subfind_density.c/93

Task=43: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.7239 MB).
task 43: endrun called with an error level of 812


 106 2980 0            Father    221.9056   26805.4894  force_treeallocate()/forcetree.c/4827
 106 2981 0           Ngblist     63.4351   26868.9245  subfind_density()/subfind_density.c/78
 106 2982 0         Dist2list    126.8702   26995.7947  subfind_density()/subfind_density.c/79
  21 2984 0      DataNodeList     58.1818   27075.7271  subfind_density()/subfind_density.c/89
  21 2985 0              Left    126.8517   27202.5789  subfind_density()/subfind_density.c/91
  21 2986 0             Right    126.8517   27329.4306  subfind_density()/subfind_density.c/92
  21 2987 0              Todo     15.8565   27345.2871  subfind_density()/subfind_density.c/93
  21 2988 0             DM_Vx    126.8517   27472.1388  subfind_density()/subfind_density.c/95
  21 2989 0             DM_Vy    126.8517   27598.9906  subfind_density()/subfind_density.c/96
  21 2990 0             DM_Vz    126.8517   27725.8423  subfind_density()/subfind_density.c/97
 116 2988 0             DM_Vx    126.6873   27468.0473  subfind_density()/subfind_density.c/95
 116 2989 0             DM_Vy    126.6873   27594.7347  subfind_density()/subfind_density.c/96
 116 2990 0             DM_Vz    126.6873   27721.4220  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 43
 106 2983 0    DataIndexTable     21.8182   27017.6128  subfind_density()/subfind_density.c/87
 106 2984 0      DataNodeList     58.1818   27075.7946  subfind_density()/subfind_density.c/89
 106 2985 0              Left    126.8702   27202.6648  subfind_density()/subfind_density.c/91
  21 2991 0    	  DensDataGet    116.1312   27841.9735  subfind_density()/subfind_density.c/161
  21 2992 0  	  DensDataResult     87.0984   27929.0720  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 116 2991 0    	  DensDataGet    109.5423   27830.9643  subfind_density()/subfind_density.c/161
 106 2986 0             Right    126.8702   27329.5350  subfind_density()/subfind_density.c/92
 106 2987 0              Todo     15.8588   27345.3938  subfind_density()/subfind_density.c/93
 106 2988 0             DM_Vx    126.8702   27472.2640  subfind_density()/subfind_density.c/95

Task=21: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.928 MB).
task 21: endrun called with an error level of 812


 116 2992 0  	  DensDataResult     82.1567   27913.1211  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 106 2989 0             DM_Vy    126.8702   27599.1342  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 21

Task=116: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=86.8789 MB).
task 116: endrun called with an error level of 812


 106 2990 0             DM_Vz    126.8702   27726.0044  subfind_density()/subfind_density.c/97
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 116
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 106 2991 0    	  DensDataGet    115.4509   27841.4554  subfind_density()/subfind_density.c/161
 120 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 120 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 120 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 120 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 120 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   7 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   7 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   7 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   7 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   7 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   7 2973 0         FOF_PList    380.6402   18861.4968  fof_fof()/fof.c/249
   7 2974 0         FOF_GList    634.4004   19495.8971  fof_fof()/fof.c/337
   7 2975 0             Group     10.9643   19506.8614  fof_fof()/fof.c/380
   7 2976 0   DomainNodeIndex      0.3498   19507.2112  force_treeallocate()/forcetree.c/4800
   7 2977 0        Nodes_base   3737.6087   23244.8199  force_treeallocate()/forcetree.c/4803
   7 2978 0     Extnodes_base   3114.6739   26359.4939  force_treeallocate()/forcetree.c/4811
   7 2979 0          Nextnode    222.3054   26581.7992  force_treeallocate()/forcetree.c/4820
   7 2980 0            Father    221.9056   26803.7048  force_treeallocate()/forcetree.c/4827
 106 2992 0  	  DensDataResult     86.5882   27928.0435  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 120 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 120 2973 0         FOF_PList    380.4128   18861.2693  fof_fof()/fof.c/249
   7 2981 0           Ngblist     63.4400   26867.1449  subfind_density()/subfind_density.c/78
   7 2982 0         Dist2list    126.8801   26994.0249  subfind_density()/subfind_density.c/79
   7 2983 0    DataIndexTable     21.8182   27015.8431  subfind_density()/subfind_density.c/87
   7 2984 0      DataNodeList     58.1818   27074.0249  subfind_density()/subfind_density.c/89

Task=106: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=71.9565 MB).
task 106: endrun called with an error level of 812


 120 2974 0         FOF_GList    634.0213   19495.2906  fof_fof()/fof.c/337
 120 2975 0             Group     13.2169   19508.5075  fof_fof()/fof.c/380
   7 2985 0              Left    126.8801   27200.9050  subfind_density()/subfind_density.c/91
   7 2986 0             Right    126.8801   27327.7850  subfind_density()/subfind_density.c/92
   7 2987 0              Todo     15.8600   27343.6451  subfind_density()/subfind_density.c/93
   7 2988 0             DM_Vx    126.8801   27470.5251  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 106
 120 2976 0   DomainNodeIndex      0.3498   19508.8574  force_treeallocate()/forcetree.c/4800
 120 2977 0        Nodes_base   3737.6087   23246.4661  force_treeallocate()/forcetree.c/4803
   7 2989 0             DM_Vy    126.8801   27597.4052  subfind_density()/subfind_density.c/96
   7 2990 0             DM_Vz    126.8801   27724.2853  subfind_density()/subfind_density.c/97
   7 2991 0    	  DensDataGet    120.5001   27844.7854  subfind_density()/subfind_density.c/161
 120 2978 0     Extnodes_base   3114.6739   26361.1400  force_treeallocate()/forcetree.c/4811
 120 2979 0          Nextnode    222.3054   26583.4454  force_treeallocate()/forcetree.c/4820
   7 2992 0  	  DensDataResult     90.3751   27935.1605  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=7: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.8395 MB).
task 7: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 120 2980 0            Father    221.9056   26805.3509  force_treeallocate()/forcetree.c/4827
 120 2981 0           Ngblist     63.4021   26868.7531  subfind_density()/subfind_density.c/78
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 7
 123 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 123 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 123 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 123 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 123 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 123 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 123 2973 0         FOF_PList    380.4981   18861.3546  fof_fof()/fof.c/249
 123 2974 0         FOF_GList    634.1635   19495.5182  fof_fof()/fof.c/337
 123 2975 0             Group     12.1212   19507.6394  fof_fof()/fof.c/380
 123 2976 0   DomainNodeIndex      0.3498   19507.9892  force_treeallocate()/forcetree.c/4800
 123 2977 0        Nodes_base   3737.6087   23245.5979  force_treeallocate()/forcetree.c/4803
 123 2978 0     Extnodes_base   3114.6739   26360.2718  force_treeallocate()/forcetree.c/4811
 123 2979 0          Nextnode    222.3054   26582.5772  force_treeallocate()/forcetree.c/4820
 123 2980 0            Father    221.9056   26804.4828  force_treeallocate()/forcetree.c/4827
 123 2981 0           Ngblist     63.4164   26867.8991  subfind_density()/subfind_density.c/78
 120 2982 0         Dist2list    126.8043   26995.5573  subfind_density()/subfind_density.c/79
   4 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   4 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   4 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   4 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   4 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   4 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   4 2973 0         FOF_PList    380.4005   18861.2570  fof_fof()/fof.c/249
   4 2974 0         FOF_GList    634.0009   19495.2579  fof_fof()/fof.c/337
   4 2975 0             Group     13.7098   19508.9677  fof_fof()/fof.c/380
   4 2976 0   DomainNodeIndex      0.3498   19509.3175  force_treeallocate()/forcetree.c/4800
 123 2982 0         Dist2list    126.8327   26994.7318  subfind_density()/subfind_density.c/79
 123 2983 0    DataIndexTable     21.8182   27016.5500  subfind_density()/subfind_density.c/87
 123 2984 0      DataNodeList     58.1818   27074.7318  subfind_density()/subfind_density.c/89
 123 2985 0              Left    126.8327   27201.5645  subfind_density()/subfind_density.c/91
 123 2986 0             Right    126.8327   27328.3972  subfind_density()/subfind_density.c/92
 120 2983 0    DataIndexTable     21.8182   27017.3755  subfind_density()/subfind_density.c/87
   4 2977 0        Nodes_base   3737.6087   23246.9262  force_treeallocate()/forcetree.c/4803
   4 2978 0     Extnodes_base   3114.6739   26361.6001  force_treeallocate()/forcetree.c/4811
 123 2987 0              Todo     15.8541   27344.2513  subfind_density()/subfind_density.c/93
 123 2988 0             DM_Vx    126.8327   27471.0840  subfind_density()/subfind_density.c/95
 123 2989 0             DM_Vy    126.8327   27597.9167  subfind_density()/subfind_density.c/96
 120 2984 0      DataNodeList     58.1818   27075.5573  subfind_density()/subfind_density.c/89
   4 2979 0          Nextnode    222.3054   26583.9055  force_treeallocate()/forcetree.c/4820
   4 2980 0            Father    221.9056   26805.8111  force_treeallocate()/forcetree.c/4827
   4 2981 0           Ngblist     63.4001   26869.2112  subfind_density()/subfind_density.c/78
 123 2990 0             DM_Vz    126.8327   27724.7494  subfind_density()/subfind_density.c/97
 123 2991 0    	  DensDataGet    127.3906   27852.1400  subfind_density()/subfind_density.c/161
 123 2992 0  	  DensDataResult     95.5429   27947.6829  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 120 2985 0              Left    126.8043   27202.3616  subfind_density()/subfind_density.c/91
   4 2982 0         Dist2list    126.8002   26996.0113  subfind_density()/subfind_density.c/79
   4 2983 0    DataIndexTable     21.8182   27017.8295  subfind_density()/subfind_density.c/87
   4 2984 0      DataNodeList     58.1818   27076.0113  subfind_density()/subfind_density.c/89

Task=123: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=52.3171 MB).
task 123: endrun called with an error level of 812


 120 2986 0             Right    126.8043   27329.1658  subfind_density()/subfind_density.c/92
   4 2985 0              Left    126.8002   27202.8115  subfind_density()/subfind_density.c/91
   4 2986 0             Right    126.8002   27329.6116  subfind_density()/subfind_density.c/92
   4 2987 0              Todo     15.8500   27345.4617  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 123
 120 2987 0              Todo     15.8505   27345.0163  subfind_density()/subfind_density.c/93
   4 2988 0             DM_Vx    126.8002   27472.2618  subfind_density()/subfind_density.c/95
 120 2988 0             DM_Vx    126.8043   27471.8206  subfind_density()/subfind_density.c/95
   4 2989 0             DM_Vy    126.8002   27599.0620  subfind_density()/subfind_density.c/96
 120 2989 0             DM_Vy    126.8043   27598.6249  subfind_density()/subfind_density.c/96
   4 2990 0             DM_Vz    126.8002   27725.8622  subfind_density()/subfind_density.c/97
 120 2990 0             DM_Vz    126.8043   27725.4291  subfind_density()/subfind_density.c/97
   4 2991 0    	  DensDataGet    121.1255   27846.9877  subfind_density()/subfind_density.c/161
 120 2991 0    	  DensDataGet    122.3913   27847.8204  subfind_density()/subfind_density.c/161
   4 2992 0  	  DensDataResult     90.8442   27937.8319  subfind_density()/subfind_density.c/201
---------------------------------------------------------------------------------------- 120 2992 0  	  DensDataResult     91.7935   27939.6139  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------


Task=120: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=60.3861 MB).

Task=4: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=62.1681 MB).
task 120: endrun called with an error level of 812


task 4: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 120
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 4
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 120:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 120: endrun called with an error level of 124


application called MPI_Abort(MPI_COMM_WORLD, 124) - process 120
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  89 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  89 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  89 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  89 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  89 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  89 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  89 2973 0         FOF_PList    380.4806   18861.3371  fof_fof()/fof.c/249
  89 2974 0         FOF_GList    634.1343   19495.4713  fof_fof()/fof.c/337
  89 2975 0             Group     12.3339   19507.8053  fof_fof()/fof.c/380
  89 2976 0   DomainNodeIndex      0.3498   19508.1551  force_treeallocate()/forcetree.c/4800
  89 2977 0        Nodes_base   3737.6087   23245.7638  force_treeallocate()/forcetree.c/4803
  89 2978 0     Extnodes_base   3114.6739   26360.4377  force_treeallocate()/forcetree.c/4811
  89 2979 0          Nextnode    222.3054   26582.7431  force_treeallocate()/forcetree.c/4820
  89 2980 0            Father    221.9056   26804.6487  force_treeallocate()/forcetree.c/4827
  89 2981 0           Ngblist     63.4134   26868.0621  subfind_density()/subfind_density.c/78
  89 2982 0         Dist2list    126.8269   26994.8889  subfind_density()/subfind_density.c/79
  89 2983 0    DataIndexTable     21.8182   27016.7071  subfind_density()/subfind_density.c/87
  89 2984 0      DataNodeList     58.1818   27074.8889  subfind_density()/subfind_density.c/89
  89 2985 0              Left    126.8269   27201.7158  subfind_density()/subfind_density.c/91
  89 2986 0             Right    126.8269   27328.5426  subfind_density()/subfind_density.c/92
  89 2987 0              Todo     15.8534   27344.3960  subfind_density()/subfind_density.c/93
  89 2988 0             DM_Vx    126.8269   27471.2228  subfind_density()/subfind_density.c/95
  89 2989 0             DM_Vy    126.8269   27598.0497  subfind_density()/subfind_density.c/96
  89 2990 0             DM_Vz    126.8269   27724.8765  subfind_density()/subfind_density.c/97
  89 2991 0    	  DensDataGet    118.0106   27842.8872  subfind_density()/subfind_density.c/161
  89 2992 0  	  DensDataResult     88.5080   27931.3951  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=89: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=68.6049 MB).
task 89: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  88 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  88 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  88 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  88 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  88 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  62 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  62 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  62 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  62 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  62 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  62 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  62 2973 0         FOF_PList    380.3164   18861.1729  fof_fof()/fof.c/249
  62 2974 0         FOF_GList    633.8607   19495.0336  fof_fof()/fof.c/337
  62 2975 0             Group     12.1990   19507.2326  fof_fof()/fof.c/380
  62 2976 0   DomainNodeIndex      0.3498   19507.5825  force_treeallocate()/forcetree.c/4800
  62 2977 0        Nodes_base   3737.6087   23245.1912  force_treeallocate()/forcetree.c/4803
  65 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  65 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  65 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  65 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  65 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  65 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  65 2973 0         FOF_PList    380.4161   18861.2726  fof_fof()/fof.c/249
  65 2974 0         FOF_GList    634.0268   19495.2994  fof_fof()/fof.c/337
  65 2975 0             Group     12.9420   19508.2414  fof_fof()/fof.c/380
  65 2976 0   DomainNodeIndex      0.3498   19508.5912  force_treeallocate()/forcetree.c/4800
  65 2977 0        Nodes_base   3737.6087   23246.1999  force_treeallocate()/forcetree.c/4803
  65 2978 0     Extnodes_base   3114.6739   26360.8738  force_treeallocate()/forcetree.c/4811
  65 2979 0          Nextnode    222.3054   26583.1792  force_treeallocate()/forcetree.c/4820
  88 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  88 2973 0         FOF_PList    380.1358   18860.9923  fof_fof()/fof.c/249
  88 2974 0         FOF_GList    633.5596   19494.5519  fof_fof()/fof.c/337
  62 2978 0     Extnodes_base   3114.6739   26359.8651  force_treeallocate()/forcetree.c/4811
  62 2979 0          Nextnode    222.3054   26582.1704  force_treeallocate()/forcetree.c/4820
  65 2980 0            Father    221.9056   26805.0848  force_treeallocate()/forcetree.c/4827
  65 2981 0           Ngblist     63.4027   26868.4874  subfind_density()/subfind_density.c/78
  65 2982 0         Dist2list    126.8054   26995.2928  subfind_density()/subfind_density.c/79
  65 2983 0    DataIndexTable     21.8182   27017.1110  subfind_density()/subfind_density.c/87
  65 2984 0      DataNodeList     58.1818   27075.2928  subfind_density()/subfind_density.c/89
  88 2975 0             Group     11.7653   19506.3172  fof_fof()/fof.c/380
  88 2976 0   DomainNodeIndex      0.3498   19506.6670  force_treeallocate()/forcetree.c/4800
  62 2980 0            Father    221.9056   26804.0760  force_treeallocate()/forcetree.c/4827
  62 2981 0           Ngblist     63.3861   26867.4621  subfind_density()/subfind_density.c/78
  62 2982 0         Dist2list    126.7721   26994.2342  subfind_density()/subfind_density.c/79
  65 2985 0              Left    126.8054   27202.0981  subfind_density()/subfind_density.c/91
  65 2986 0             Right    126.8054   27328.9035  subfind_density()/subfind_density.c/92
  65 2987 0              Todo     15.8507   27344.7542  subfind_density()/subfind_density.c/93
  88 2977 0        Nodes_base   3737.6087   23244.2757  force_treeallocate()/forcetree.c/4803
  88 2978 0     Extnodes_base   3114.6739   26358.9497  force_treeallocate()/forcetree.c/4811
  88 2979 0          Nextnode    222.3054   26581.2550  force_treeallocate()/forcetree.c/4820
  62 2983 0    DataIndexTable     21.8182   27016.0524  subfind_density()/subfind_density.c/87
  62 2984 0      DataNodeList     58.1818   27074.2342  subfind_density()/subfind_density.c/89
  65 2988 0             DM_Vx    126.8054   27471.5595  subfind_density()/subfind_density.c/95
  65 2989 0             DM_Vy    126.8054   27598.3649  subfind_density()/subfind_density.c/96
  65 2990 0             DM_Vz    126.8054   27725.1702  subfind_density()/subfind_density.c/97
  88 2980 0            Father    221.9056   26803.1606  force_treeallocate()/forcetree.c/4827
  88 2981 0           Ngblist     63.3560   26866.5166  subfind_density()/subfind_density.c/78
  88 2982 0         Dist2list    126.7119   26993.2285  subfind_density()/subfind_density.c/79
  62 2985 0              Left    126.7721   27201.0063  subfind_density()/subfind_density.c/91
  62 2986 0             Right    126.7721   27327.7785  subfind_density()/subfind_density.c/92
  62 2987 0              Todo     15.8465   27343.6250  subfind_density()/subfind_density.c/93
  65 2991 0    	  DensDataGet    130.7161   27855.8863  subfind_density()/subfind_density.c/161
  65 2992 0  	  DensDataResult     98.0370   27953.9233  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=65: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=46.0767 MB).
task 65: endrun called with an error level of 812


  88 2983 0    DataIndexTable     21.8182   27015.0467  subfind_density()/subfind_density.c/87
  62 2988 0             DM_Vx    126.7721   27470.3971  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 65
  88 2984 0      DataNodeList     58.1818   27073.2285  subfind_density()/subfind_density.c/89
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  62 2989 0             DM_Vy    126.7721   27597.1693  subfind_density()/subfind_density.c/96
  62 2990 0             DM_Vz    126.7721   27723.9414  subfind_density()/subfind_density.c/97
  62 2991 0    	  DensDataGet    115.3383   27839.2797  subfind_density()/subfind_density.c/161
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  88 2985 0              Left    126.7119   27199.9404  subfind_density()/subfind_density.c/91
 107 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 107 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 107 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 107 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 107 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 107 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
 107 2973 0         FOF_PList    380.4677   18861.3242  fof_fof()/fof.c/249
 107 2974 0         FOF_GList    634.1128   19495.4370  fof_fof()/fof.c/337
 107 2975 0             Group     11.7383   19507.1754  fof_fof()/fof.c/380
 107 2976 0   DomainNodeIndex      0.3498   19507.5252  force_treeallocate()/forcetree.c/4800
 107 2977 0        Nodes_base   3737.6087   23245.1339  force_treeallocate()/forcetree.c/4803
 107 2978 0     Extnodes_base   3114.6739   26359.8078  force_treeallocate()/forcetree.c/4811
  62 2992 0  	  DensDataResult     86.5037   27925.7834  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  20 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  20 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  20 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  20 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  20 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  20 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  20 2973 0         FOF_PList    380.6287   18861.4852  fof_fof()/fof.c/249
  88 2986 0             Right    126.7119   27326.6523  subfind_density()/subfind_density.c/92
 107 2979 0          Nextnode    222.3054   26582.1132  force_treeallocate()/forcetree.c/4820
 107 2980 0            Father    221.9056   26804.0188  force_treeallocate()/forcetree.c/4827
 107 2981 0           Ngblist     63.4113   26867.4301  subfind_density()/subfind_density.c/78
 107 2982 0         Dist2list    126.8226   26994.2526  subfind_density()/subfind_density.c/79

Task=62: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.2166 MB).
  20 2974 0         FOF_GList    634.3812   19495.8664  fof_fof()/fof.c/337
  20 2975 0             Group     14.1279   19509.9944  fof_fof()/fof.c/380
  88 2987 0              Todo     15.8390   27342.4913  subfind_density()/subfind_density.c/93
 107 2983 0    DataIndexTable     21.8182   27016.0708  subfind_density()/subfind_density.c/87
 107 2984 0      DataNodeList     58.1818   27074.2526  subfind_density()/subfind_density.c/89
 107 2985 0              Left    126.8226   27201.0752  subfind_density()/subfind_density.c/91
 107 2986 0             Right    126.8226   27327.8977  subfind_density()/subfind_density.c/92
task 62: endrun called with an error level of 812


  20 2976 0   DomainNodeIndex      0.3498   19510.3442  force_treeallocate()/forcetree.c/4800
  20 2977 0        Nodes_base   3737.6087   23247.9529  force_treeallocate()/forcetree.c/4803
  20 2978 0     Extnodes_base   3114.6739   26362.6268  force_treeallocate()/forcetree.c/4811
  88 2988 0             DM_Vx    126.7119   27469.2032  subfind_density()/subfind_density.c/95
 107 2987 0              Todo     15.8528   27343.7505  subfind_density()/subfind_density.c/93
 107 2988 0             DM_Vx    126.8226   27470.5731  subfind_density()/subfind_density.c/95
 107 2989 0             DM_Vy    126.8226   27597.3957  subfind_density()/subfind_density.c/96
 107 2990 0             DM_Vz    126.8226   27724.2182  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 62
  20 2979 0          Nextnode    222.3054   26584.9322  force_treeallocate()/forcetree.c/4820
  20 2980 0            Father    221.9056   26806.8378  force_treeallocate()/forcetree.c/4827
  20 2981 0           Ngblist     63.4381   26870.2759  subfind_density()/subfind_density.c/78
  88 2989 0             DM_Vy    126.7119   27595.9152  subfind_density()/subfind_density.c/96
 107 2991 0    	  DensDataGet    119.0081   27843.2264  subfind_density()/subfind_density.c/161
 107 2992 0  	  DensDataResult     89.2561   27932.4824  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=107: Not enough memory in mymalloc_fullinfo() to allocate 87.2726 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=67.5176 MB).
  20 2982 0         Dist2list    126.8762   26997.1521  subfind_density()/subfind_density.c/79
  20 2983 0    DataIndexTable     21.8182   27018.9703  subfind_density()/subfind_density.c/87
  88 2990 0             DM_Vz    126.7119   27722.6271  subfind_density()/subfind_density.c/97
task 107: endrun called with an error level of 812


  20 2984 0      DataNodeList     58.1818   27077.1521  subfind_density()/subfind_density.c/89
  20 2985 0              Left    126.8762   27204.0283  subfind_density()/subfind_density.c/91
  88 2991 0    	  DensDataGet    113.9442   27836.5712  subfind_density()/subfind_density.c/161
  88 2992 0  	  DensDataResult     85.4581   27922.0294  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=88: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.9706 MB).
task 88: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 107
  20 2986 0             Right    126.8762   27330.9045  subfind_density()/subfind_density.c/92
  20 2987 0              Todo     15.8595   27346.7641  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 88
  20 2988 0             DM_Vx    126.8762   27473.6403  subfind_density()/subfind_density.c/95
task 88: endrun called with an error level of 124


  20 2989 0             DM_Vy    126.8762   27600.5166  subfind_density()/subfind_density.c/96
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 88:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 88
  20 2990 0             DM_Vz    126.8762   27727.3928  subfind_density()/subfind_density.c/97
  20 2991 0    	  DensDataGet    136.6639   27864.0567  subfind_density()/subfind_density.c/161
  20 2992 0  	  DensDataResult    102.4979   27966.5546  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=20: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=33.4454 MB).
task 20: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 20
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
 122 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
 122 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
 122 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
 122 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
 122 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
 122 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   5 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
   5 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
   5 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
   5 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
   5 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
   5 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
   5 2973 0         FOF_PList    380.4788   18861.3354  fof_fof()/fof.c/249
   5 2974 0         FOF_GList    634.1314   19495.4668  fof_fof()/fof.c/337
   5 2975 0             Group     12.6514   19508.1182  fof_fof()/fof.c/380
   5 2976 0   DomainNodeIndex      0.3498   19508.4680  force_treeallocate()/forcetree.c/4800
 122 2973 0         FOF_PList    380.5814   18861.4379  fof_fof()/fof.c/249
 122 2974 0         FOF_GList    634.3023   19495.7403  fof_fof()/fof.c/337
 122 2975 0             Group     13.4390   19509.1792  fof_fof()/fof.c/380
   5 2977 0        Nodes_base   3737.6087   23246.0767  force_treeallocate()/forcetree.c/4803
   5 2978 0     Extnodes_base   3114.6739   26360.7506  force_treeallocate()/forcetree.c/4811
   5 2979 0          Nextnode    222.3054   26583.0560  force_treeallocate()/forcetree.c/4820
   5 2980 0            Father    221.9056   26804.9616  force_treeallocate()/forcetree.c/4827
   5 2981 0           Ngblist     63.4131   26868.3747  subfind_density()/subfind_density.c/78
 122 2976 0   DomainNodeIndex      0.3498   19509.5290  force_treeallocate()/forcetree.c/4800
 122 2977 0        Nodes_base   3737.6087   23247.1377  force_treeallocate()/forcetree.c/4803
 122 2978 0     Extnodes_base   3114.6739   26361.8117  force_treeallocate()/forcetree.c/4811
   5 2982 0         Dist2list    126.8263   26995.2010  subfind_density()/subfind_density.c/79
   5 2983 0    DataIndexTable     21.8182   27017.0192  subfind_density()/subfind_density.c/87
   5 2984 0      DataNodeList     58.1818   27075.2010  subfind_density()/subfind_density.c/89
 122 2979 0          Nextnode    222.3054   26584.1170  force_treeallocate()/forcetree.c/4820
 122 2980 0            Father    221.9056   26806.0226  force_treeallocate()/forcetree.c/4827
 122 2981 0           Ngblist     63.4302   26869.4529  subfind_density()/subfind_density.c/78
   5 2985 0              Left    126.8263   27202.0273  subfind_density()/subfind_density.c/91
   5 2986 0             Right    126.8263   27328.8535  subfind_density()/subfind_density.c/92
   5 2987 0              Todo     15.8533   27344.7068  subfind_density()/subfind_density.c/93
   5 2988 0             DM_Vx    126.8263   27471.5331  subfind_density()/subfind_density.c/95
 122 2982 0         Dist2list    126.8605   26996.3133  subfind_density()/subfind_density.c/79
 122 2983 0    DataIndexTable     21.8182   27018.1315  subfind_density()/subfind_density.c/87
 122 2984 0      DataNodeList     58.1818   27076.3133  subfind_density()/subfind_density.c/89
   5 2989 0             DM_Vy    126.8263   27598.3594  subfind_density()/subfind_density.c/96
   5 2990 0             DM_Vz    126.8263   27725.1857  subfind_density()/subfind_density.c/97
   5 2991 0    	  DensDataGet    139.7997   27864.9854  subfind_density()/subfind_density.c/161
 122 2985 0              Left    126.8605   27203.1738  subfind_density()/subfind_density.c/91
 122 2986 0             Right    126.8605   27330.0342  subfind_density()/subfind_density.c/92
   5 2992 0  	  DensDataResult    104.8498   27969.8352  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=5: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=30.1648 MB).
task 5: endrun called with an error level of 812


 122 2987 0              Todo     15.8576   27345.8918  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 5
 122 2988 0             DM_Vx    126.8605   27472.7523  subfind_density()/subfind_density.c/95
 122 2989 0             DM_Vy    126.8605   27599.6127  subfind_density()/subfind_density.c/96
 122 2990 0             DM_Vz    126.8605   27726.4732  subfind_density()/subfind_density.c/97
 122 2991 0    	  DensDataGet    118.0997   27844.5729  subfind_density()/subfind_density.c/161
 122 2992 0  	  DensDataResult     88.5748   27933.1477  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=122: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.8523 MB).
task 122: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 122
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  81 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  81 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  81 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  81 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  81 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  81 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  81 2973 0         FOF_PList    380.4751   18861.3316  fof_fof()/fof.c/249
  81 2974 0         FOF_GList    634.1252   19495.4568  fof_fof()/fof.c/337
  81 2975 0             Group     10.9197   19506.3765  fof_fof()/fof.c/380
  81 2976 0   DomainNodeIndex      0.3498   19506.7263  force_treeallocate()/forcetree.c/4800
  81 2977 0        Nodes_base   3737.6087   23244.3350  force_treeallocate()/forcetree.c/4803
  81 2978 0     Extnodes_base   3114.6739   26359.0089  force_treeallocate()/forcetree.c/4811
  81 2979 0          Nextnode    222.3054   26581.3143  force_treeallocate()/forcetree.c/4820
  81 2980 0            Father    221.9056   26803.2199  force_treeallocate()/forcetree.c/4827
  81 2981 0           Ngblist     63.4125   26866.6324  subfind_density()/subfind_density.c/78
  81 2982 0         Dist2list    126.8250   26993.4575  subfind_density()/subfind_density.c/79
  81 2983 0    DataIndexTable     21.8182   27015.2756  subfind_density()/subfind_density.c/87
  81 2984 0      DataNodeList     58.1818   27073.4574  subfind_density()/subfind_density.c/89
  81 2985 0              Left    126.8250   27200.2825  subfind_density()/subfind_density.c/91
  81 2986 0             Right    126.8250   27327.1075  subfind_density()/subfind_density.c/92
  81 2987 0              Todo     15.8531   27342.9606  subfind_density()/subfind_density.c/93
  81 2988 0             DM_Vx    126.8250   27469.7857  subfind_density()/subfind_density.c/95
  81 2989 0             DM_Vy    126.8250   27596.6107  subfind_density()/subfind_density.c/96
  81 2990 0             DM_Vz    126.8250   27723.4357  subfind_density()/subfind_density.c/97
  81 2991 0    	  DensDataGet    113.5031   27836.9388  subfind_density()/subfind_density.c/161
  81 2992 0  	  DensDataResult     85.1273   27922.0661  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=81: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=77.9339 MB).
task 81: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 81
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  93 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  93 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  93 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  93 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  93 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  93 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  93 2973 0         FOF_PList    380.3887   18861.2452  fof_fof()/fof.c/249
  93 2974 0         FOF_GList    633.9811   19495.2263  fof_fof()/fof.c/337
  93 2975 0             Group     11.9469   19507.1732  fof_fof()/fof.c/380
  93 2976 0   DomainNodeIndex      0.3498   19507.5231  force_treeallocate()/forcetree.c/4800
  93 2977 0        Nodes_base   3737.6087   23245.1318  force_treeallocate()/forcetree.c/4803
  93 2978 0     Extnodes_base   3114.6739   26359.8057  force_treeallocate()/forcetree.c/4811
  93 2979 0          Nextnode    222.3054   26582.1111  force_treeallocate()/forcetree.c/4820
  93 2980 0            Father    221.9056   26804.0166  force_treeallocate()/forcetree.c/4827
  93 2981 0           Ngblist     63.3981   26867.4148  subfind_density()/subfind_density.c/78
  93 2982 0         Dist2list    126.7962   26994.2110  subfind_density()/subfind_density.c/79
  93 2983 0    DataIndexTable     21.8182   27016.0292  subfind_density()/subfind_density.c/87
  93 2984 0      DataNodeList     58.1818   27074.2110  subfind_density()/subfind_density.c/89
  93 2985 0              Left    126.7962   27201.0072  subfind_density()/subfind_density.c/91
  93 2986 0             Right    126.7962   27327.8034  subfind_density()/subfind_density.c/92
  93 2987 0              Todo     15.8495   27343.6529  subfind_density()/subfind_density.c/93
  93 2988 0             DM_Vx    126.7962   27470.4492  subfind_density()/subfind_density.c/95
  93 2989 0             DM_Vy    126.7962   27597.2454  subfind_density()/subfind_density.c/96
  93 2990 0             DM_Vz    126.7962   27724.0416  subfind_density()/subfind_density.c/97
  93 2991 0    	  DensDataGet    110.3883   27834.4299  subfind_density()/subfind_density.c/161
  93 2992 0  	  DensDataResult     82.7912   27917.2212  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=93: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.7788 MB).
task 93: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 93
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  82 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  82 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  82 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  82 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  82 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  82 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  82 2973 0         FOF_PList    380.5150   18861.3716  fof_fof()/fof.c/249
  82 2974 0         FOF_GList    634.1917   19495.5633  fof_fof()/fof.c/337
  82 2975 0             Group     13.3933   19508.9566  fof_fof()/fof.c/380
  82 2976 0   DomainNodeIndex      0.3498   19509.3064  force_treeallocate()/forcetree.c/4800
  82 2977 0        Nodes_base   3737.6087   23246.9152  force_treeallocate()/forcetree.c/4803
  82 2978 0     Extnodes_base   3114.6739   26361.5891  force_treeallocate()/forcetree.c/4811
  82 2979 0          Nextnode    222.3054   26583.8944  force_treeallocate()/forcetree.c/4820
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  35 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  35 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  35 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  35 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  35 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  35 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  35 2973 0         FOF_PList    380.5775   18861.4340  fof_fof()/fof.c/249
  35 2974 0         FOF_GList    634.2958   19495.7298  fof_fof()/fof.c/337
  35 2975 0             Group     12.4688   19508.1986  fof_fof()/fof.c/380
  35 2976 0   DomainNodeIndex      0.3498   19508.5485  force_treeallocate()/forcetree.c/4800
  35 2977 0        Nodes_base   3737.6087   23246.1572  force_treeallocate()/forcetree.c/4803
  35 2978 0     Extnodes_base   3114.6739   26360.8311  force_treeallocate()/forcetree.c/4811
  82 2980 0            Father    221.9056   26805.8000  force_treeallocate()/forcetree.c/4827
  82 2981 0           Ngblist     63.4192   26869.2192  subfind_density()/subfind_density.c/78
  45 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  45 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  45 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  45 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  45 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  45 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  45 2973 0         FOF_PList    380.4109   18861.2674  fof_fof()/fof.c/249
  45 2974 0         FOF_GList    634.0181   19495.2856  fof_fof()/fof.c/337
  45 2975 0             Group     11.6522   19506.9378  fof_fof()/fof.c/380
  45 2976 0   DomainNodeIndex      0.3498   19507.2876  force_treeallocate()/forcetree.c/4800
  45 2977 0        Nodes_base   3737.6087   23244.8963  force_treeallocate()/forcetree.c/4803
  45 2978 0     Extnodes_base   3114.6739   26359.5702  force_treeallocate()/forcetree.c/4811
  45 2979 0          Nextnode    222.3054   26581.8756  force_treeallocate()/forcetree.c/4820
  45 2980 0            Father    221.9056   26803.7812  force_treeallocate()/forcetree.c/4827
  45 2981 0           Ngblist     63.4018   26867.1830  subfind_density()/subfind_density.c/78
  45 2982 0         Dist2list    126.8036   26993.9866  subfind_density()/subfind_density.c/79
  35 2979 0          Nextnode    222.3054   26583.1365  force_treeallocate()/forcetree.c/4820
  35 2980 0            Father    221.9056   26805.0420  force_treeallocate()/forcetree.c/4827
  35 2981 0           Ngblist     63.4296   26868.4716  subfind_density()/subfind_density.c/78
  35 2982 0         Dist2list    126.8592   26995.3308  subfind_density()/subfind_density.c/79
  35 2983 0    DataIndexTable     21.8182   27017.1490  subfind_density()/subfind_density.c/87
  82 2982 0         Dist2list    126.8383   26996.0576  subfind_density()/subfind_density.c/79
  82 2983 0    DataIndexTable     21.8182   27017.8757  subfind_density()/subfind_density.c/87
  45 2983 0    DataIndexTable     21.8182   27015.8048  subfind_density()/subfind_density.c/87
  45 2984 0      DataNodeList     58.1818   27073.9866  subfind_density()/subfind_density.c/89
  45 2985 0              Left    126.8036   27200.7902  subfind_density()/subfind_density.c/91
  45 2986 0             Right    126.8036   27327.5938  subfind_density()/subfind_density.c/92
  45 2987 0              Todo     15.8505   27343.4443  subfind_density()/subfind_density.c/93
  35 2984 0      DataNodeList     58.1818   27075.3307  subfind_density()/subfind_density.c/89
  35 2985 0              Left    126.8592   27202.1899  subfind_density()/subfind_density.c/91
  35 2986 0             Right    126.8592   27329.0491  subfind_density()/subfind_density.c/92
  82 2984 0      DataNodeList     58.1818   27076.0575  subfind_density()/subfind_density.c/89
  82 2985 0              Left    126.8383   27202.8959  subfind_density()/subfind_density.c/91
  82 2986 0             Right    126.8383   27329.7342  subfind_density()/subfind_density.c/92
  45 2988 0             DM_Vx    126.8036   27470.2479  subfind_density()/subfind_density.c/95
  45 2989 0             DM_Vy    126.8036   27597.0516  subfind_density()/subfind_density.c/96
  45 2990 0             DM_Vz    126.8036   27723.8552  subfind_density()/subfind_density.c/97
  35 2987 0              Todo     15.8574   27344.9065  subfind_density()/subfind_density.c/93
  35 2988 0             DM_Vx    126.8592   27471.7656  subfind_density()/subfind_density.c/95
  35 2989 0             DM_Vy    126.8592   27598.6248  subfind_density()/subfind_density.c/96
  82 2987 0              Todo     15.8548   27345.5890  subfind_density()/subfind_density.c/93
  82 2988 0             DM_Vx    126.8383   27472.4274  subfind_density()/subfind_density.c/95
  82 2989 0             DM_Vy    126.8383   27599.2657  subfind_density()/subfind_density.c/96
  45 2991 0    	  DensDataGet    125.9121   27849.7673  subfind_density()/subfind_density.c/161
  45 2992 0  	  DensDataResult     94.4341   27944.2014  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=45: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=55.7986 MB).
  35 2990 0             DM_Vz    126.8592   27725.4840  subfind_density()/subfind_density.c/97
  35 2991 0    	  DensDataGet    109.9716   27835.4555  subfind_density()/subfind_density.c/161
  35 2992 0  	  DensDataResult     82.4787   27917.9342  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  82 2990 0             DM_Vz    126.8383   27726.1041  subfind_density()/subfind_density.c/97
task 45: endrun called with an error level of 812



Task=35: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=82.0658 MB).
  82 2991 0    	  DensDataGet    118.6509   27844.7550  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 45
task 35: endrun called with an error level of 812


  82 2992 0  	  DensDataResult     88.9882   27933.7432  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------application called MPI_Abort(MPI_COMM_WORLD, 812) - process 35


Task=82: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.2568 MB).
task 82: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 82
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  64 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  64 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  64 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  64 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  64 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  63 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  63 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  63 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  63 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  63 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  63 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  63 2973 0         FOF_PList    380.3908   18861.2473  fof_fof()/fof.c/249
  63 2974 0         FOF_GList    633.9847   19495.2320  fof_fof()/fof.c/337
  63 2975 0             Group     11.1936   19506.4256  fof_fof()/fof.c/380
  63 2976 0   DomainNodeIndex      0.3498   19506.7754  force_treeallocate()/forcetree.c/4800
  63 2977 0        Nodes_base   3737.6087   23244.3841  force_treeallocate()/forcetree.c/4803
  63 2978 0     Extnodes_base   3114.6739   26359.0581  force_treeallocate()/forcetree.c/4811
  64 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  64 2973 0         FOF_PList    380.3296   18861.1862  fof_fof()/fof.c/249
  63 2979 0          Nextnode    222.3054   26581.3634  force_treeallocate()/forcetree.c/4820
  63 2980 0            Father    221.9056   26803.2690  force_treeallocate()/forcetree.c/4827
  63 2981 0           Ngblist     63.3985   26866.6675  subfind_density()/subfind_density.c/78
  63 2982 0         Dist2list    126.7969   26993.4644  subfind_density()/subfind_density.c/79
  64 2974 0         FOF_GList    633.8827   19495.0689  fof_fof()/fof.c/337
  64 2975 0             Group     13.6641   19508.7330  fof_fof()/fof.c/380
  64 2976 0   DomainNodeIndex      0.3498   19509.0828  force_treeallocate()/forcetree.c/4800
  63 2983 0    DataIndexTable     21.8182   27015.2826  subfind_density()/subfind_density.c/87
  63 2984 0      DataNodeList     58.1818   27073.4644  subfind_density()/subfind_density.c/89
  63 2985 0              Left    126.7969   27200.2613  subfind_density()/subfind_density.c/91
  64 2977 0        Nodes_base   3737.6087   23246.6915  force_treeallocate()/forcetree.c/4803
  64 2978 0     Extnodes_base   3114.6739   26361.3654  force_treeallocate()/forcetree.c/4811
  64 2979 0          Nextnode    222.3054   26583.6708  force_treeallocate()/forcetree.c/4820
  63 2986 0             Right    126.7969   27327.0583  subfind_density()/subfind_density.c/92
  63 2987 0              Todo     15.8496   27342.9079  subfind_density()/subfind_density.c/93
  63 2988 0             DM_Vx    126.7969   27469.7048  subfind_density()/subfind_density.c/95
  64 2980 0            Father    221.9056   26805.5764  force_treeallocate()/forcetree.c/4827
  64 2981 0           Ngblist     63.3883   26868.9647  subfind_density()/subfind_density.c/78
  64 2982 0         Dist2list    126.7765   26995.7412  subfind_density()/subfind_density.c/79
  63 2989 0             DM_Vy    126.7969   27596.5018  subfind_density()/subfind_density.c/96
  63 2990 0             DM_Vz    126.7969   27723.2987  subfind_density()/subfind_density.c/97
  63 2991 0    	  DensDataGet    123.8773   27847.1760  subfind_density()/subfind_density.c/161
  64 2983 0    DataIndexTable     21.8182   27017.5594  subfind_density()/subfind_density.c/87
  63 2992 0  	  DensDataResult     92.9080   27940.0840  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=63: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=59.916 MB).
task 63: endrun called with an error level of 812


  64 2984 0      DataNodeList     58.1818   27075.7412  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 63
  64 2985 0              Left    126.7765   27202.5177  subfind_density()/subfind_density.c/91
  64 2986 0             Right    126.7765   27329.2943  subfind_density()/subfind_density.c/92
  64 2987 0              Todo     15.8471   27345.1413  subfind_density()/subfind_density.c/93
  64 2988 0             DM_Vx    126.7765   27471.9179  subfind_density()/subfind_density.c/95
  64 2989 0             DM_Vy    126.7765   27598.6944  subfind_density()/subfind_density.c/96
  64 2990 0             DM_Vz    126.7765   27725.4710  subfind_density()/subfind_density.c/97
  64 2991 0    	  DensDataGet    115.0898   27840.5608  subfind_density()/subfind_density.c/161
  64 2992 0  	  DensDataResult     86.3173   27926.8781  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=64: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=73.1219 MB).
task 64: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 64
task 64: endrun called with an error level of 124


 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 64:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 64
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  70 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  70 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  70 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  70 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  70 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  70 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  70 2973 0         FOF_PList    379.8263   18860.6829  fof_fof()/fof.c/249
  70 2974 0         FOF_GList    633.0439   19493.7268  fof_fof()/fof.c/337
  70 2975 0             Group     12.3827   19506.1095  fof_fof()/fof.c/380
  70 2976 0   DomainNodeIndex      0.3498   19506.4593  force_treeallocate()/forcetree.c/4800
  70 2977 0        Nodes_base   3737.6087   23244.0680  force_treeallocate()/forcetree.c/4803
  70 2978 0     Extnodes_base   3114.6739   26358.7419  force_treeallocate()/forcetree.c/4811
  70 2979 0          Nextnode    222.3054   26581.0473  force_treeallocate()/forcetree.c/4820
  57 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  57 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  57 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  57 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  57 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  57 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  57 2973 0         FOF_PList    380.5895   18861.4461  fof_fof()/fof.c/249
  57 2974 0         FOF_GList    634.3159   19495.7619  fof_fof()/fof.c/337
  57 2975 0             Group     11.2994   19507.0614  fof_fof()/fof.c/380
  57 2976 0   DomainNodeIndex      0.3498   19507.4112  force_treeallocate()/forcetree.c/4800
  57 2977 0        Nodes_base   3737.6087   23245.0199  force_treeallocate()/forcetree.c/4803
  57 2978 0     Extnodes_base   3114.6739   26359.6938  force_treeallocate()/forcetree.c/4811
  57 2979 0          Nextnode    222.3054   26581.9992  force_treeallocate()/forcetree.c/4820
  57 2980 0            Father    221.9056   26803.9048  force_treeallocate()/forcetree.c/4827
  57 2981 0           Ngblist     63.4316   26867.3364  subfind_density()/subfind_density.c/78
  57 2982 0         Dist2list    126.8632   26994.1995  subfind_density()/subfind_density.c/79
  57 2983 0    DataIndexTable     21.8182   27016.0177  subfind_density()/subfind_density.c/87
  70 2980 0            Father    221.9056   26802.9529  force_treeallocate()/forcetree.c/4827
  70 2981 0           Ngblist     63.3044   26866.2573  subfind_density()/subfind_density.c/78
  57 2984 0      DataNodeList     58.1818   27074.1995  subfind_density()/subfind_density.c/89
  57 2985 0              Left    126.8632   27201.0627  subfind_density()/subfind_density.c/91
  57 2986 0             Right    126.8632   27327.9258  subfind_density()/subfind_density.c/92
  57 2987 0              Todo     15.8579   27343.7838  subfind_density()/subfind_density.c/93
  57 2988 0             DM_Vx    126.8632   27470.6469  subfind_density()/subfind_density.c/95
  57 2989 0             DM_Vy    126.8632   27597.5101  subfind_density()/subfind_density.c/96
  70 2982 0         Dist2list    126.6088   26992.8660  subfind_density()/subfind_density.c/79
  70 2983 0    DataIndexTable     21.8182   27014.6842  subfind_density()/subfind_density.c/87
  70 2984 0      DataNodeList     58.1818   27072.8660  subfind_density()/subfind_density.c/89
  57 2990 0             DM_Vz    126.8632   27724.3733  subfind_density()/subfind_density.c/97
  57 2991 0    	  DensDataGet    137.4203   27861.7936  subfind_density()/subfind_density.c/161
  57 2992 0  	  DensDataResult    103.0653   27964.8589  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  70 2985 0              Left    126.6088   27199.4748  subfind_density()/subfind_density.c/91
  70 2986 0             Right    126.6088   27326.0836  subfind_density()/subfind_density.c/92

Task=57: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=35.1411 MB).
task 57: endrun called with an error level of 812


  70 2987 0              Todo     15.8261   27341.9097  subfind_density()/subfind_density.c/93
  70 2988 0             DM_Vx    126.6088   27468.5184  subfind_density()/subfind_density.c/95
  70 2989 0             DM_Vy    126.6088   27595.1272  subfind_density()/subfind_density.c/96
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 57
  70 2990 0             DM_Vz    126.6088   27721.7360  subfind_density()/subfind_density.c/97
  70 2991 0    	  DensDataGet    121.1615   27842.8975  subfind_density()/subfind_density.c/161
  70 2992 0  	  DensDataResult     90.8711   27933.7686  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=70: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=66.2314 MB).
task 70: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 70
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  78 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  78 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  78 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  78 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  78 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  78 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  78 2973 0         FOF_PList    380.3442   18861.2007  fof_fof()/fof.c/249
  78 2974 0         FOF_GList    633.9069   19495.1076  fof_fof()/fof.c/337
  78 2975 0             Group     11.6636   19506.7713  fof_fof()/fof.c/380
  78 2976 0   DomainNodeIndex      0.3498   19507.1211  force_treeallocate()/forcetree.c/4800
  78 2977 0        Nodes_base   3737.6087   23244.7298  force_treeallocate()/forcetree.c/4803
  78 2978 0     Extnodes_base   3114.6739   26359.4037  force_treeallocate()/forcetree.c/4811
  78 2979 0          Nextnode    222.3054   26581.7091  force_treeallocate()/forcetree.c/4820
  78 2980 0            Father    221.9056   26803.6147  force_treeallocate()/forcetree.c/4827
  78 2981 0           Ngblist     63.3907   26867.0054  subfind_density()/subfind_density.c/78
  78 2982 0         Dist2list    126.7814   26993.7868  subfind_density()/subfind_density.c/79
  78 2983 0    DataIndexTable     21.8182   27015.6049  subfind_density()/subfind_density.c/87
  78 2984 0      DataNodeList     58.1818   27073.7867  subfind_density()/subfind_density.c/89
  78 2985 0              Left    126.7814   27200.5681  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  78 2986 0             Right    126.7814   27327.3495  subfind_density()/subfind_density.c/92
  74 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  74 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  74 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  74 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  74 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  74 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  74 2973 0         FOF_PList    380.6215   18861.4781  fof_fof()/fof.c/249
  74 2974 0         FOF_GList    634.3692   19495.8473  fof_fof()/fof.c/337
  74 2975 0             Group     13.0270   19508.8743  fof_fof()/fof.c/380
  74 2976 0   DomainNodeIndex      0.3498   19509.2241  force_treeallocate()/forcetree.c/4800
  78 2987 0              Todo     15.8477   27343.1972  subfind_density()/subfind_density.c/93
  74 2977 0        Nodes_base   3737.6087   23246.8328  force_treeallocate()/forcetree.c/4803
  74 2978 0     Extnodes_base   3114.6739   26361.5067  force_treeallocate()/forcetree.c/4811
  74 2979 0          Nextnode    222.3054   26583.8121  force_treeallocate()/forcetree.c/4820
  74 2980 0            Father    221.9056   26805.7177  force_treeallocate()/forcetree.c/4827
  78 2988 0             DM_Vx    126.7814   27469.9786  subfind_density()/subfind_density.c/95
  74 2981 0           Ngblist     63.4369   26869.1546  subfind_density()/subfind_density.c/78
  74 2982 0         Dist2list    126.8738   26996.0285  subfind_density()/subfind_density.c/79
  74 2983 0    DataIndexTable     21.8182   27017.8466  subfind_density()/subfind_density.c/87
  78 2989 0             DM_Vy    126.7814   27596.7599  subfind_density()/subfind_density.c/96
  78 2990 0             DM_Vz    126.7814   27723.5413  subfind_density()/subfind_density.c/97
  78 2991 0    	  DensDataGet    109.8545   27833.3958  subfind_density()/subfind_density.c/161
  74 2984 0      DataNodeList     58.1818   27076.0284  subfind_density()/subfind_density.c/89
  74 2985 0              Left    126.8738   27202.9023  subfind_density()/subfind_density.c/91
  74 2986 0             Right    126.8738   27329.7761  subfind_density()/subfind_density.c/92
  78 2992 0  	  DensDataResult     82.3909   27915.7867  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  74 2987 0              Todo     15.8592   27345.6353  subfind_density()/subfind_density.c/93
  74 2988 0             DM_Vx    126.8738   27472.5092  subfind_density()/subfind_density.c/95
  74 2989 0             DM_Vy    126.8738   27599.3830  subfind_density()/subfind_density.c/96

Task=78: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.2133 MB).
  74 2990 0             DM_Vz    126.8738   27726.2569  subfind_density()/subfind_density.c/97
task 78: endrun called with an error level of 812


  74 2991 0    	  DensDataGet    118.8520   27845.1088  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 78
  74 2992 0  	  DensDataResult     89.1390   27934.2478  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=74: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=65.7522 MB).
task 74: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 74
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  60 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  60 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  60 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  60 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  60 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  60 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  60 2973 0         FOF_PList    380.6385   18861.4951  fof_fof()/fof.c/249
  60 2974 0         FOF_GList    634.3975   19495.8926  fof_fof()/fof.c/337
  67 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  67 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  67 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  67 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  67 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  67 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  67 2973 0         FOF_PList    380.5063   18861.3628  fof_fof()/fof.c/249
  67 2974 0         FOF_GList    634.1771   19495.5399  fof_fof()/fof.c/337
  67 2975 0             Group     10.9197   19506.4596  fof_fof()/fof.c/380
  67 2976 0   DomainNodeIndex      0.3498   19506.8094  force_treeallocate()/forcetree.c/4800
  67 2977 0        Nodes_base   3737.6087   23244.4181  force_treeallocate()/forcetree.c/4803
  67 2978 0     Extnodes_base   3114.6739   26359.0920  force_treeallocate()/forcetree.c/4811
  67 2979 0          Nextnode    222.3054   26581.3974  force_treeallocate()/forcetree.c/4820
  67 2980 0            Father    221.9056   26803.3030  force_treeallocate()/forcetree.c/4827
  67 2981 0           Ngblist     63.4177   26866.7207  subfind_density()/subfind_density.c/78
  67 2982 0         Dist2list    126.8354   26993.5561  subfind_density()/subfind_density.c/79
  60 2975 0             Group     12.5829   19508.4756  fof_fof()/fof.c/380
  60 2976 0   DomainNodeIndex      0.3498   19508.8254  force_treeallocate()/forcetree.c/4800
  60 2977 0        Nodes_base   3737.6087   23246.4341  force_treeallocate()/forcetree.c/4803
  67 2983 0    DataIndexTable     21.8182   27015.3743  subfind_density()/subfind_density.c/87
  67 2984 0      DataNodeList     58.1818   27073.5561  subfind_density()/subfind_density.c/89
  67 2985 0              Left    126.8354   27200.3915  subfind_density()/subfind_density.c/91
  67 2986 0             Right    126.8354   27327.2269  subfind_density()/subfind_density.c/92
  60 2978 0     Extnodes_base   3114.6739   26361.1080  force_treeallocate()/forcetree.c/4811
  60 2979 0          Nextnode    222.3054   26583.4134  force_treeallocate()/forcetree.c/4820
  60 2980 0            Father    221.9056   26805.3190  force_treeallocate()/forcetree.c/4827
  60 2981 0           Ngblist     63.4398   26868.7587  subfind_density()/subfind_density.c/78
  67 2987 0              Todo     15.8544   27343.0813  subfind_density()/subfind_density.c/93
  67 2988 0             DM_Vx    126.8354   27469.9167  subfind_density()/subfind_density.c/95
  60 2982 0         Dist2list    126.8795   26995.6382  subfind_density()/subfind_density.c/79
  60 2983 0    DataIndexTable     21.8182   27017.4564  subfind_density()/subfind_density.c/87
  60 2984 0      DataNodeList     58.1818   27075.6382  subfind_density()/subfind_density.c/89
  67 2989 0             DM_Vy    126.8354   27596.7522  subfind_density()/subfind_density.c/96
  67 2990 0             DM_Vz    126.8354   27723.5876  subfind_density()/subfind_density.c/97
  67 2991 0    	  DensDataGet    125.1129   27848.7004  subfind_density()/subfind_density.c/161
  60 2985 0              Left    126.8795   27202.5177  subfind_density()/subfind_density.c/91
  60 2986 0             Right    126.8795   27329.3972  subfind_density()/subfind_density.c/92
  67 2992 0  	  DensDataResult     93.8346   27942.5351  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  60 2987 0              Todo     15.8599   27345.2571  subfind_density()/subfind_density.c/93
  60 2988 0             DM_Vx    126.8795   27472.1367  subfind_density()/subfind_density.c/95
  60 2989 0             DM_Vy    126.8795   27599.0162  subfind_density()/subfind_density.c/96

Task=67: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=57.4649 MB).
task 67: endrun called with an error level of 812


  60 2990 0             DM_Vz    126.8795   27725.8957  subfind_density()/subfind_density.c/97
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 67
  60 2991 0    	  DensDataGet    108.1631   27834.0588  subfind_density()/subfind_density.c/161
  60 2992 0  	  DensDataResult     81.1223   27915.1811  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=60: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.8189 MB).
task 60: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 60
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  33 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  33 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  33 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  33 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  33 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  33 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  33 2973 0         FOF_PList    380.6525   18861.5090  fof_fof()/fof.c/249
  33 2974 0         FOF_GList    634.4208   19495.9298  fof_fof()/fof.c/337
  33 2975 0             Group     11.0297   19506.9594  fof_fof()/fof.c/380
  33 2976 0   DomainNodeIndex      0.3498   19507.3093  force_treeallocate()/forcetree.c/4800
  33 2977 0        Nodes_base   3737.6087   23244.9180  force_treeallocate()/forcetree.c/4803
  33 2978 0     Extnodes_base   3114.6739   26359.5919  force_treeallocate()/forcetree.c/4811
  94 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  94 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  94 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  94 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  94 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  94 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  94 2973 0         FOF_PList    379.4966   18860.3531  fof_fof()/fof.c/249
  94 2974 0         FOF_GList    632.4943   19492.8474  fof_fof()/fof.c/337
  94 2975 0             Group     12.7230   19505.5705  fof_fof()/fof.c/380
  94 2976 0   DomainNodeIndex      0.3498   19505.9203  force_treeallocate()/forcetree.c/4800
  94 2977 0        Nodes_base   3737.6087   23243.5290  force_treeallocate()/forcetree.c/4803
  33 2979 0          Nextnode    222.3054   26581.8973  force_treeallocate()/forcetree.c/4820
  33 2980 0            Father    221.9056   26803.8028  force_treeallocate()/forcetree.c/4827
  33 2981 0           Ngblist     63.4421   26867.2449  subfind_density()/subfind_density.c/78
  33 2982 0         Dist2list    126.8842   26994.1291  subfind_density()/subfind_density.c/79
  94 2978 0     Extnodes_base   3114.6739   26358.2029  force_treeallocate()/forcetree.c/4811
  94 2979 0          Nextnode    222.3054   26580.5083  force_treeallocate()/forcetree.c/4820
  94 2980 0            Father    221.9056   26802.4139  force_treeallocate()/forcetree.c/4827
  94 2981 0           Ngblist     63.2494   26865.6633  subfind_density()/subfind_density.c/78
  33 2983 0    DataIndexTable     21.8182   27015.9473  subfind_density()/subfind_density.c/87
  33 2984 0      DataNodeList     58.1818   27074.1290  subfind_density()/subfind_density.c/89
  94 2982 0         Dist2list    126.4989   26992.1622  subfind_density()/subfind_density.c/79
  94 2983 0    DataIndexTable     21.8182   27013.9803  subfind_density()/subfind_density.c/87
  94 2984 0      DataNodeList     58.1818   27072.1621  subfind_density()/subfind_density.c/89
  33 2985 0              Left    126.8842   27201.0132  subfind_density()/subfind_density.c/91
  33 2986 0             Right    126.8842   27327.8974  subfind_density()/subfind_density.c/92
  33 2987 0              Todo     15.8605   27343.7579  subfind_density()/subfind_density.c/93
  94 2985 0              Left    126.4989   27198.6610  subfind_density()/subfind_density.c/91
  94 2986 0             Right    126.4989   27325.1599  subfind_density()/subfind_density.c/92
  33 2988 0             DM_Vx    126.8842   27470.6420  subfind_density()/subfind_density.c/95
  33 2989 0             DM_Vy    126.8842   27597.5262  subfind_density()/subfind_density.c/96
  94 2987 0              Todo     15.8124   27340.9722  subfind_density()/subfind_density.c/93
  94 2988 0             DM_Vx    126.4989   27467.4711  subfind_density()/subfind_density.c/95
  94 2989 0             DM_Vy    126.4989   27593.9699  subfind_density()/subfind_density.c/96
  33 2990 0             DM_Vz    126.8842   27724.4103  subfind_density()/subfind_density.c/97
  94 2990 0             DM_Vz    126.4989   27720.4688  subfind_density()/subfind_density.c/97
  94 2991 0    	  DensDataGet    114.0271   27834.4959  subfind_density()/subfind_density.c/161
  33 2991 0    	  DensDataGet    111.8839   27836.2943  subfind_density()/subfind_density.c/161
  94 2992 0  	  DensDataResult     85.5203   27920.0162  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  33 2992 0  	  DensDataResult     83.9129   27920.2072  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=94: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.9838 MB).
task 94: endrun called with an error level of 812



Task=33: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=79.7928 MB).
task 33: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 94
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 33
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  73 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  73 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  73 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  73 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  73 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  73 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  73 2973 0         FOF_PList    380.4879   18861.3445  fof_fof()/fof.c/249
  73 2974 0         FOF_GList    634.1465   19495.4910  fof_fof()/fof.c/337
  73 2975 0             Group     11.8369   19507.3279  fof_fof()/fof.c/380
  73 2976 0   DomainNodeIndex      0.3498   19507.6777  force_treeallocate()/forcetree.c/4800
  73 2977 0        Nodes_base   3737.6087   23245.2864  force_treeallocate()/forcetree.c/4803
  73 2978 0     Extnodes_base   3114.6739   26359.9604  force_treeallocate()/forcetree.c/4811
  73 2979 0          Nextnode    222.3054   26582.2657  force_treeallocate()/forcetree.c/4820
  73 2980 0            Father    221.9056   26804.1713  force_treeallocate()/forcetree.c/4827
  73 2981 0           Ngblist     63.4147   26867.5860  subfind_density()/subfind_density.c/78
  54 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  54 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  54 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  54 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  54 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  54 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  54 2973 0         FOF_PList    380.3927   18861.2492  fof_fof()/fof.c/249
  54 2974 0         FOF_GList    633.9878   19495.2371  fof_fof()/fof.c/337
  54 2975 0             Group     13.2989   19508.5360  fof_fof()/fof.c/380
  54 2976 0   DomainNodeIndex      0.3498   19508.8858  force_treeallocate()/forcetree.c/4800
  54 2977 0        Nodes_base   3737.6087   23246.4945  force_treeallocate()/forcetree.c/4803
  32 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  32 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  32 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  32 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  32 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  95 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  95 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  95 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  95 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  95 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  95 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  95 2973 0         FOF_PList    379.6775   18860.5341  fof_fof()/fof.c/249
  95 2974 0         FOF_GList    632.7959   19493.3299  fof_fof()/fof.c/337
  95 2975 0             Group     13.5832   19506.9131  fof_fof()/fof.c/380
  95 2976 0   DomainNodeIndex      0.3498   19507.2629  force_treeallocate()/forcetree.c/4800
  95 2977 0        Nodes_base   3737.6087   23244.8716  force_treeallocate()/forcetree.c/4803
  95 2978 0     Extnodes_base   3114.6739   26359.5456  force_treeallocate()/forcetree.c/4811
  95 2979 0          Nextnode    222.3054   26581.8509  force_treeallocate()/forcetree.c/4820
  73 2982 0         Dist2list    126.8293   26994.4153  subfind_density()/subfind_density.c/79
  73 2983 0    DataIndexTable     21.8182   27016.2335  subfind_density()/subfind_density.c/87
  73 2984 0      DataNodeList     58.1818   27074.4152  subfind_density()/subfind_density.c/89
  73 2985 0              Left    126.8293   27201.2446  subfind_density()/subfind_density.c/91
  73 2986 0             Right    126.8293   27328.0739  subfind_density()/subfind_density.c/92
  54 2978 0     Extnodes_base   3114.6739   26361.1684  force_treeallocate()/forcetree.c/4811
  54 2979 0          Nextnode    222.3054   26583.4738  force_treeallocate()/forcetree.c/4820
  54 2980 0            Father    221.9056   26805.3794  force_treeallocate()/forcetree.c/4827
  54 2981 0           Ngblist     63.3988   26868.7782  subfind_density()/subfind_density.c/78
  32 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  32 2973 0         FOF_PList    380.6271   18861.4836  fof_fof()/fof.c/249
  95 2980 0            Father    221.9056   26803.7565  force_treeallocate()/forcetree.c/4827
  95 2981 0           Ngblist     63.2796   26867.0361  subfind_density()/subfind_density.c/78
  95 2982 0         Dist2list    126.5592   26993.5953  subfind_density()/subfind_density.c/79
  73 2987 0              Todo     15.8537   27343.9275  subfind_density()/subfind_density.c/93
  73 2988 0             DM_Vx    126.8293   27470.7568  subfind_density()/subfind_density.c/95
  73 2989 0             DM_Vy    126.8293   27597.5861  subfind_density()/subfind_density.c/96
  54 2982 0         Dist2list    126.7976   26995.5757  subfind_density()/subfind_density.c/79
  54 2983 0    DataIndexTable     21.8182   27017.3939  subfind_density()/subfind_density.c/87
  54 2984 0      DataNodeList     58.1818   27075.5757  subfind_density()/subfind_density.c/89
  32 2974 0         FOF_GList    634.3784   19495.8620  fof_fof()/fof.c/337
  32 2975 0             Group     12.2924   19508.1544  fof_fof()/fof.c/380
  95 2983 0    DataIndexTable     21.8182   27015.4135  subfind_density()/subfind_density.c/87
  95 2984 0      DataNodeList     58.1818   27073.5952  subfind_density()/subfind_density.c/89
  95 2985 0              Left    126.5592   27200.1544  subfind_density()/subfind_density.c/91
  73 2990 0             DM_Vz    126.8293   27724.4155  subfind_density()/subfind_density.c/97
  73 2991 0    	  DensDataGet    112.4750   27836.8905  subfind_density()/subfind_density.c/161
  73 2992 0  	  DensDataResult     84.3563   27921.2468  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  54 2985 0              Left    126.7976   27202.3733  subfind_density()/subfind_density.c/91
  54 2986 0             Right    126.7976   27329.1708  subfind_density()/subfind_density.c/92
  54 2987 0              Todo     15.8497   27345.0205  subfind_density()/subfind_density.c/93
  32 2976 0   DomainNodeIndex      0.3498   19508.5043  force_treeallocate()/forcetree.c/4800
  32 2977 0        Nodes_base   3737.6087   23246.1130  force_treeallocate()/forcetree.c/4803
  32 2978 0     Extnodes_base   3114.6739   26360.7869  force_treeallocate()/forcetree.c/4811
  95 2986 0             Right    126.5592   27326.7136  subfind_density()/subfind_density.c/92
  95 2987 0              Todo     15.8199   27342.5335  subfind_density()/subfind_density.c/93
  95 2988 0             DM_Vx    126.5592   27469.0927  subfind_density()/subfind_density.c/95
  95 2989 0             DM_Vy    126.5592   27595.6518  subfind_density()/subfind_density.c/96

Task=73: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=78.7532 MB).
task 73: endrun called with an error level of 812


  54 2988 0             DM_Vx    126.7976   27471.8181  subfind_density()/subfind_density.c/95
  54 2989 0             DM_Vy    126.7976   27598.6157  subfind_density()/subfind_density.c/96
  54 2990 0             DM_Vz    126.7976   27725.4133  subfind_density()/subfind_density.c/97
  32 2979 0          Nextnode    222.3054   26583.0923  force_treeallocate()/forcetree.c/4820
  32 2980 0            Father    221.9056   26804.9978  force_treeallocate()/forcetree.c/4827
  95 2990 0             DM_Vz    126.5592   27722.2110  subfind_density()/subfind_density.c/97
  95 2991 0    	  DensDataGet    131.8331   27854.0441  subfind_density()/subfind_density.c/161
  95 2992 0  	  DensDataResult     98.8748   27952.9189  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 73
  54 2991 0    	  DensDataGet    113.3877   27838.8009  subfind_density()/subfind_density.c/161
  32 2981 0           Ngblist     63.4378   26868.4357  subfind_density()/subfind_density.c/78
  32 2982 0         Dist2list    126.8757   26995.3114  subfind_density()/subfind_density.c/79

Task=95: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=47.0811 MB).
task 95: endrun called with an error level of 812


  54 2992 0  	  DensDataResult     85.0408   27923.8417  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  32 2983 0    DataIndexTable     21.8182   27017.1296  subfind_density()/subfind_density.c/87
  32 2984 0      DataNodeList     58.1818   27075.3113  subfind_density()/subfind_density.c/89
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 95

Task=54: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.1583 MB).
task 54: endrun called with an error level of 812


  32 2985 0              Left    126.8757   27202.1870  subfind_density()/subfind_density.c/91
  32 2986 0             Right    126.8757   27329.0627  subfind_density()/subfind_density.c/92
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 54
  32 2987 0              Todo     15.8595   27344.9222  subfind_density()/subfind_density.c/93
  32 2988 0             DM_Vx    126.8757   27471.7979  subfind_density()/subfind_density.c/95
  32 2989 0             DM_Vy    126.8757   27598.6736  subfind_density()/subfind_density.c/96
  32 2990 0             DM_Vz    126.8757   27725.5492  subfind_density()/subfind_density.c/97
  32 2991 0    	  DensDataGet    131.0672   27856.6164  subfind_density()/subfind_density.c/161
  32 2992 0  	  DensDataResult     98.3004   27954.9168  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=32: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.0832 MB).
task 32: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 32
 An HDF5 error was detected, we'd better stop.
HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 32:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
task 32: endrun called with an error level of 124


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 32
  69 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  69 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  69 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  69 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  69 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  69 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  69 2973 0         FOF_PList    380.5731   18861.4296  fof_fof()/fof.c/249
  69 2974 0         FOF_GList    634.2885   19495.7182  fof_fof()/fof.c/337
  69 2975 0             Group     11.5402   19507.2583  fof_fof()/fof.c/380
  69 2976 0   DomainNodeIndex      0.3498   19507.6081  force_treeallocate()/forcetree.c/4800
  69 2977 0        Nodes_base   3737.6087   23245.2169  force_treeallocate()/forcetree.c/4803
  69 2978 0     Extnodes_base   3114.6739   26359.8908  force_treeallocate()/forcetree.c/4811
  69 2979 0          Nextnode    222.3054   26582.1961  force_treeallocate()/forcetree.c/4820
  69 2980 0            Father    221.9056   26804.1017  force_treeallocate()/forcetree.c/4827
  69 2981 0           Ngblist     63.4289   26867.5306  subfind_density()/subfind_density.c/78
  69 2982 0         Dist2list    126.8577   26994.3883  subfind_density()/subfind_density.c/79
  69 2983 0    DataIndexTable     21.8182   27016.2065  subfind_density()/subfind_density.c/87
  69 2984 0      DataNodeList     58.1818   27074.3883  subfind_density()/subfind_density.c/89
  69 2985 0              Left    126.8577   27201.2460  subfind_density()/subfind_density.c/91
  69 2986 0             Right    126.8577   27328.1037  subfind_density()/subfind_density.c/92
  69 2987 0              Todo     15.8572   27343.9609  subfind_density()/subfind_density.c/93
  69 2988 0             DM_Vx    126.8577   27470.8186  subfind_density()/subfind_density.c/95
  69 2989 0             DM_Vy    126.8577   27597.6763  subfind_density()/subfind_density.c/96
  69 2990 0             DM_Vz    126.8577   27724.5340  subfind_density()/subfind_density.c/97
  69 2991 0    	  DensDataGet    120.5909   27845.1249  subfind_density()/subfind_density.c/161
  69 2992 0  	  DensDataResult     90.4432   27935.5680  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=69: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=64.432 MB).
task 69: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 69
  61 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  61 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  61 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  61 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  61 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  61 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  61 2973 0         FOF_PList    380.5418   18861.3983  fof_fof()/fof.c/249
  61 2974 0         FOF_GList    634.2363   19495.6347  fof_fof()/fof.c/337
  61 2975 0             Group     10.9197   19506.5544  fof_fof()/fof.c/380
  61 2976 0   DomainNodeIndex      0.3498   19506.9042  force_treeallocate()/forcetree.c/4800
  61 2977 0        Nodes_base   3737.6087   23244.5129  force_treeallocate()/forcetree.c/4803
  61 2978 0     Extnodes_base   3114.6739   26359.1868  force_treeallocate()/forcetree.c/4811
  61 2979 0          Nextnode    222.3054   26581.4922  force_treeallocate()/forcetree.c/4820
  61 2980 0            Father    221.9056   26803.3978  force_treeallocate()/forcetree.c/4827
  61 2981 0           Ngblist     63.4236   26866.8214  subfind_density()/subfind_density.c/78
  61 2982 0         Dist2list    126.8473   26993.6687  subfind_density()/subfind_density.c/79
  61 2983 0    DataIndexTable     21.8182   27015.4868  subfind_density()/subfind_density.c/87
  61 2984 0      DataNodeList     58.1818   27073.6686  subfind_density()/subfind_density.c/89
  61 2985 0              Left    126.8473   27200.5159  subfind_density()/subfind_density.c/91
  61 2986 0             Right    126.8473   27327.3632  subfind_density()/subfind_density.c/92
  61 2987 0              Todo     15.8559   27343.2191  subfind_density()/subfind_density.c/93
  61 2988 0             DM_Vx    126.8473   27470.0663  subfind_density()/subfind_density.c/95
  61 2989 0             DM_Vy    126.8473   27596.9136  subfind_density()/subfind_density.c/96
  61 2990 0             DM_Vz    126.8473   27723.7609  subfind_density()/subfind_density.c/97
  61 2991 0    	  DensDataGet    109.7120   27833.4728  subfind_density()/subfind_density.c/161
  61 2992 0  	  DensDataResult     82.2840   27915.7568  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=61: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=84.2432 MB).
task 61: endrun called with an error level of 812


------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 61
  72 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  72 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  72 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  72 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  72 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  72 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  55 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  55 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  55 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  55 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  55 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  55 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  55 2973 0         FOF_PList    380.5722   18861.4287  fof_fof()/fof.c/249
  55 2974 0         FOF_GList    634.2870   19495.7157  fof_fof()/fof.c/337
  55 2975 0             Group     12.4252   19508.1410  fof_fof()/fof.c/380
  55 2976 0   DomainNodeIndex      0.3498   19508.4908  force_treeallocate()/forcetree.c/4800
  55 2977 0        Nodes_base   3737.6087   23246.0995  force_treeallocate()/forcetree.c/4803
  55 2978 0     Extnodes_base   3114.6739   26360.7734  force_treeallocate()/forcetree.c/4811
  55 2979 0          Nextnode    222.3054   26583.0788  force_treeallocate()/forcetree.c/4820
  72 2973 0         FOF_PList    380.3350   18861.1915  fof_fof()/fof.c/249
  72 2974 0         FOF_GList    633.8916   19495.0831  fof_fof()/fof.c/337
  55 2980 0            Father    221.9056   26804.9844  force_treeallocate()/forcetree.c/4827
  55 2981 0           Ngblist     63.4287   26868.4131  subfind_density()/subfind_density.c/78
  55 2982 0         Dist2list    126.8574   26995.2705  subfind_density()/subfind_density.c/79
  72 2975 0             Group     11.8338   19506.9169  fof_fof()/fof.c/380
  72 2976 0   DomainNodeIndex      0.3498   19507.2667  force_treeallocate()/forcetree.c/4800
  72 2977 0        Nodes_base   3737.6087   23244.8754  force_treeallocate()/forcetree.c/4803
  55 2983 0    DataIndexTable     21.8182   27017.0886  subfind_density()/subfind_density.c/87
  55 2984 0      DataNodeList     58.1818   27075.2704  subfind_density()/subfind_density.c/89
  55 2985 0              Left    126.8574   27202.1278  subfind_density()/subfind_density.c/91
  72 2978 0     Extnodes_base   3114.6739   26359.5493  force_treeallocate()/forcetree.c/4811
  72 2979 0          Nextnode    222.3054   26581.8547  force_treeallocate()/forcetree.c/4820
  72 2980 0            Father    221.9056   26803.7603  force_treeallocate()/forcetree.c/4827
  55 2986 0             Right    126.8574   27328.9852  subfind_density()/subfind_density.c/92
  55 2987 0              Todo     15.8572   27344.8424  subfind_density()/subfind_density.c/93
  55 2988 0             DM_Vx    126.8574   27471.6998  subfind_density()/subfind_density.c/95
  55 2989 0             DM_Vy    126.8574   27598.5572  subfind_density()/subfind_density.c/96
  72 2981 0           Ngblist     63.3892   26867.1495  subfind_density()/subfind_density.c/78
  72 2982 0         Dist2list    126.7783   26993.9278  subfind_density()/subfind_density.c/79
  72 2983 0    DataIndexTable     21.8182   27015.7460  subfind_density()/subfind_density.c/87
  55 2990 0             DM_Vz    126.8574   27725.4146  subfind_density()/subfind_density.c/97
  55 2991 0    	  DensDataGet    123.6871   27849.1017  subfind_density()/subfind_density.c/161
  55 2992 0  	  DensDataResult     92.7653   27941.8670  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  72 2984 0      DataNodeList     58.1818   27073.9277  subfind_density()/subfind_density.c/89
  72 2985 0              Left    126.7783   27200.7061  subfind_density()/subfind_density.c/91

Task=55: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=58.133 MB).
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  72 2986 0             Right    126.7783   27327.4844  subfind_density()/subfind_density.c/92
task 55: endrun called with an error level of 812


  52 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  52 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  52 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  52 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  52 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  52 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  52 2973 0         FOF_PList    380.5517   18861.4082  fof_fof()/fof.c/249
  52 2974 0         FOF_GList    634.2529   19495.6611  fof_fof()/fof.c/337
  52 2975 0             Group     13.1100   19508.7711  fof_fof()/fof.c/380
  52 2976 0   DomainNodeIndex      0.3498   19509.1210  force_treeallocate()/forcetree.c/4800
  52 2977 0        Nodes_base   3737.6087   23246.7297  force_treeallocate()/forcetree.c/4803
  52 2978 0     Extnodes_base   3114.6739   26361.4036  force_treeallocate()/forcetree.c/4811
  52 2979 0          Nextnode    222.3054   26583.7090  force_treeallocate()/forcetree.c/4820
  75 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  75 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  75 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  75 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  75 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  75 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  75 2973 0         FOF_PList    380.5113   18861.3678  fof_fof()/fof.c/249
  75 2974 0         FOF_GList    634.1854   19495.5533  fof_fof()/fof.c/337
  75 2975 0             Group     12.1586   19507.7118  fof_fof()/fof.c/380
  75 2976 0   DomainNodeIndex      0.3498   19508.0616  force_treeallocate()/forcetree.c/4800
  75 2977 0        Nodes_base   3737.6087   23245.6703  force_treeallocate()/forcetree.c/4803
  75 2978 0     Extnodes_base   3114.6739   26360.3443  force_treeallocate()/forcetree.c/4811
  72 2987 0              Todo     15.8473   27343.3317  subfind_density()/subfind_density.c/93
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 55
  52 2980 0            Father    221.9056   26805.6145  force_treeallocate()/forcetree.c/4827
  52 2981 0           Ngblist     63.4253   26869.0398  subfind_density()/subfind_density.c/78
  52 2982 0         Dist2list    126.8506   26995.8904  subfind_density()/subfind_density.c/79
  75 2979 0          Nextnode    222.3054   26582.6496  force_treeallocate()/forcetree.c/4820
  75 2980 0            Father    221.9056   26804.5552  force_treeallocate()/forcetree.c/4827
  75 2981 0           Ngblist     63.4185   26867.9738  subfind_density()/subfind_density.c/78
  75 2982 0         Dist2list    126.8371   26994.8109  subfind_density()/subfind_density.c/79
  72 2988 0             DM_Vx    126.7783   27470.1100  subfind_density()/subfind_density.c/95
  52 2983 0    DataIndexTable     21.8182   27017.7086  subfind_density()/subfind_density.c/87
  52 2984 0      DataNodeList     58.1818   27075.8904  subfind_density()/subfind_density.c/89
  52 2985 0              Left    126.8506   27202.7409  subfind_density()/subfind_density.c/91
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  75 2983 0    DataIndexTable     21.8182   27016.6290  subfind_density()/subfind_density.c/87
  75 2984 0      DataNodeList     58.1818   27074.8108  subfind_density()/subfind_density.c/89
  75 2985 0              Left    126.8371   27201.6479  subfind_density()/subfind_density.c/91
  75 2986 0             Right    126.8371   27328.4850  subfind_density()/subfind_density.c/92
  72 2989 0             DM_Vy    126.7783   27596.8883  subfind_density()/subfind_density.c/96
  52 2986 0             Right    126.8506   27329.5915  subfind_density()/subfind_density.c/92
  52 2987 0              Todo     15.8563   27345.4478  subfind_density()/subfind_density.c/93
  52 2988 0             DM_Vx    126.8506   27472.2984  subfind_density()/subfind_density.c/95
  48 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  48 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  48 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  48 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  48 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  48 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  48 2973 0         FOF_PList    380.3725   18861.2290  fof_fof()/fof.c/249
  48 2974 0         FOF_GList    633.9541   19495.1831  fof_fof()/fof.c/337
  48 2975 0             Group     12.9420   19508.1251  fof_fof()/fof.c/380
  48 2976 0   DomainNodeIndex      0.3498   19508.4749  force_treeallocate()/forcetree.c/4800
  48 2977 0        Nodes_base   3737.6087   23246.0836  force_treeallocate()/forcetree.c/4803
  48 2978 0     Extnodes_base   3114.6739   26360.7575  force_treeallocate()/forcetree.c/4811
  75 2987 0              Todo     15.8546   27344.3396  subfind_density()/subfind_density.c/93
  75 2988 0             DM_Vx    126.8371   27471.1767  subfind_density()/subfind_density.c/95
  75 2989 0             DM_Vy    126.8371   27598.0138  subfind_density()/subfind_density.c/96
  72 2990 0             DM_Vz    126.7783   27723.6666  subfind_density()/subfind_density.c/97
  52 2989 0             DM_Vy    126.8506   27599.1490  subfind_density()/subfind_density.c/96
  52 2990 0             DM_Vz    126.8506   27725.9995  subfind_density()/subfind_density.c/97
  48 2979 0          Nextnode    222.3054   26583.0629  force_treeallocate()/forcetree.c/4820
  48 2980 0            Father    221.9056   26804.9685  force_treeallocate()/forcetree.c/4827
  48 2981 0           Ngblist     63.3954   26868.3639  subfind_density()/subfind_density.c/78
  48 2982 0         Dist2list    126.7908   26995.1547  subfind_density()/subfind_density.c/79
  75 2990 0             DM_Vz    126.8371   27724.8509  subfind_density()/subfind_density.c/97
  75 2991 0    	  DensDataGet    113.7380   27838.5890  subfind_density()/subfind_density.c/161
  75 2992 0  	  DensDataResult     85.3035   27923.8925  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  72 2991 0    	  DensDataGet    117.4096   27841.0762  subfind_density()/subfind_density.c/161
  52 2991 0    	  DensDataGet    134.5198   27860.5193  subfind_density()/subfind_density.c/161
  52 2992 0  	  DensDataResult    100.8898   27961.4092  subfind_density()/subfind_density.c/201
  48 2983 0    DataIndexTable     21.8182   27016.9729  subfind_density()/subfind_density.c/87
  48 2984 0      DataNodeList     58.1818   27075.1547  subfind_density()/subfind_density.c/89
  48 2985 0              Left    126.7908   27201.9455  subfind_density()/subfind_density.c/91

Task=75: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=76.1075 MB).
task 75: endrun called with an error level of 812


  72 2992 0  	  DensDataResult     88.0572   27929.1335  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
  48 2986 0             Right    126.7908   27328.7364  subfind_density()/subfind_density.c/92
  48 2987 0              Todo     15.8489   27344.5852  subfind_density()/subfind_density.c/93
  48 2988 0             DM_Vx    126.7908   27471.3760  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 75

Task=72: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=70.8665 MB).

Task=52: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=38.5908 MB).
  48 2989 0             DM_Vy    126.7908   27598.1669  subfind_density()/subfind_density.c/96
  48 2990 0             DM_Vz    126.7908   27724.9577  subfind_density()/subfind_density.c/97
task 72: endrun called with an error level of 812


task 52: endrun called with an error level of 812


  48 2991 0    	  DensDataGet    113.9694   27838.9271  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 72
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 52
  48 2992 0  	  DensDataResult     85.4771   27924.4042  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
 An HDF5 error was detected, we'd better stop.

Task=48: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=75.5958 MB).
task 48: endrun called with an error level of 812


HDF5-DIAG: Error detected in HDF5 (1.8.12) application called MPI_Abort(MPI_COMM_WORLD, 812) - process 48
MPI-process 72:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
 An HDF5 error was detected, we'd better stop.
    minor: Can't get value
task 48: endrun called with an error level of 124


task 72: endrun called with an error level of 124


HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 48:
  #000: ../../hdf5-1.8.12/src/H5E.c line 1615 in H5Eget_auto2(): wrong API function, H5Eset_auto1 has been called
    major: Error API
    minor: Can't get value
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 48
application called MPI_Abort(MPI_COMM_WORLD, 124) - process 72
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  44 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  44 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  44 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  44 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  44 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  44 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  44 2973 0         FOF_PList    380.5062   18861.3627  fof_fof()/fof.c/249
  44 2974 0         FOF_GList    634.1770   19495.5397  fof_fof()/fof.c/337
  83 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  83 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  83 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  83 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  83 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  83 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  83 2973 0         FOF_PList    380.5084   18861.3649  fof_fof()/fof.c/249
  83 2974 0         FOF_GList    634.1807   19495.5456  fof_fof()/fof.c/337
  83 2975 0             Group     12.1627   19507.7083  fof_fof()/fof.c/380
  83 2976 0   DomainNodeIndex      0.3498   19508.0582  force_treeallocate()/forcetree.c/4800
  83 2977 0        Nodes_base   3737.6087   23245.6669  force_treeallocate()/forcetree.c/4803
  83 2978 0     Extnodes_base   3114.6739   26360.3408  force_treeallocate()/forcetree.c/4811
  83 2979 0          Nextnode    222.3054   26582.6462  force_treeallocate()/forcetree.c/4820
  83 2980 0            Father    221.9056   26804.5517  force_treeallocate()/forcetree.c/4827
  83 2981 0           Ngblist     63.4181   26867.9698  subfind_density()/subfind_density.c/78
  44 2975 0             Group     11.4385   19506.9782  fof_fof()/fof.c/380
  44 2976 0   DomainNodeIndex      0.3498   19507.3280  force_treeallocate()/forcetree.c/4800
  44 2977 0        Nodes_base   3737.6087   23244.9367  force_treeallocate()/forcetree.c/4803
  83 2982 0         Dist2list    126.8361   26994.8059  subfind_density()/subfind_density.c/79
  83 2983 0    DataIndexTable     21.8182   27016.6241  subfind_density()/subfind_density.c/87
  83 2984 0      DataNodeList     58.1818   27074.8059  subfind_density()/subfind_density.c/89
  83 2985 0              Left    126.8361   27201.6421  subfind_density()/subfind_density.c/91
  44 2978 0     Extnodes_base   3114.6739   26359.6106  force_treeallocate()/forcetree.c/4811
  44 2979 0          Nextnode    222.3054   26581.9160  force_treeallocate()/forcetree.c/4820
  44 2980 0            Father    221.9056   26803.8216  force_treeallocate()/forcetree.c/4827
  83 2986 0             Right    126.8361   27328.4782  subfind_density()/subfind_density.c/92
  83 2987 0              Todo     15.8545   27344.3327  subfind_density()/subfind_density.c/93
  83 2988 0             DM_Vx    126.8361   27471.1688  subfind_density()/subfind_density.c/95
  83 2989 0             DM_Vy    126.8361   27598.0050  subfind_density()/subfind_density.c/96
  44 2981 0           Ngblist     63.4177   26867.2393  subfind_density()/subfind_density.c/78
  44 2982 0         Dist2list    126.8354   26994.0747  subfind_density()/subfind_density.c/79
  44 2983 0    DataIndexTable     21.8182   27015.8929  subfind_density()/subfind_density.c/87
  83 2990 0             DM_Vz    126.8361   27724.8411  subfind_density()/subfind_density.c/97
  83 2991 0    	  DensDataGet    121.1232   27845.9643  subfind_density()/subfind_density.c/161
  83 2992 0  	  DensDataResult     90.8424   27936.8068  subfind_density()/subfind_density.c/201
  44 2984 0      DataNodeList     58.1818   27074.0746  subfind_density()/subfind_density.c/89
  44 2985 0              Left    126.8354   27200.9100  subfind_density()/subfind_density.c/91
  44 2986 0             Right    126.8354   27327.7454  subfind_density()/subfind_density.c/92
----------------------------------------------------------------------------------------

Task=83: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=63.1932 MB).
task 83: endrun called with an error level of 812


  44 2987 0              Todo     15.8544   27343.5999  subfind_density()/subfind_density.c/93
  44 2988 0             DM_Vx    126.8354   27470.4353  subfind_density()/subfind_density.c/95
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 83
  44 2989 0             DM_Vy    126.8354   27597.2707  subfind_density()/subfind_density.c/96
  44 2990 0             DM_Vz    126.8354   27724.1061  subfind_density()/subfind_density.c/97
  44 2991 0    	  DensDataGet    129.4926   27853.5987  subfind_density()/subfind_density.c/161
  44 2992 0  	  DensDataResult     97.1195   27950.7181  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------

Task=44: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=49.2819 MB).
task 44: endrun called with an error level of 812


  68 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  68 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  68 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  68 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  68 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  68 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  59 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  59 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  59 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  59 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  59 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 44
  68 2973 0         FOF_PList    380.6505   18861.5071  fof_fof()/fof.c/249
  68 2974 0         FOF_GList    634.4175   19495.9246  fof_fof()/fof.c/337
  68 2975 0             Group     12.7593   19508.6839  fof_fof()/fof.c/380
  59 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  59 2973 0         FOF_PList    379.3995   18860.2561  fof_fof()/fof.c/249
  59 2974 0         FOF_GList    632.3325   19492.5886  fof_fof()/fof.c/337
  59 2975 0             Group     11.7280   19504.3166  fof_fof()/fof.c/380
  59 2976 0   DomainNodeIndex      0.3498   19504.6664  force_treeallocate()/forcetree.c/4800
  59 2977 0        Nodes_base   3737.6087   23242.2751  force_treeallocate()/forcetree.c/4803
  59 2978 0     Extnodes_base   3114.6739   26356.9490  force_treeallocate()/forcetree.c/4811
  59 2979 0          Nextnode    222.3054   26579.2544  force_treeallocate()/forcetree.c/4820
  59 2980 0            Father    221.9056   26801.1600  force_treeallocate()/forcetree.c/4827
  59 2981 0           Ngblist     63.2333   26864.3932  subfind_density()/subfind_density.c/78
  59 2982 0         Dist2list    126.4665   26990.8597  subfind_density()/subfind_density.c/79
  59 2983 0    DataIndexTable     21.8182   27012.6779  subfind_density()/subfind_density.c/87
  59 2984 0      DataNodeList     58.1818   27070.8597  subfind_density()/subfind_density.c/89
  59 2985 0              Left    126.4665   27197.3262  subfind_density()/subfind_density.c/91
  59 2986 0             Right    126.4665   27323.7927  subfind_density()/subfind_density.c/92
  59 2987 0              Todo     15.8083   27339.6010  subfind_density()/subfind_density.c/93
  59 2988 0             DM_Vx    126.4665   27466.0675  subfind_density()/subfind_density.c/95
  59 2989 0             DM_Vy    126.4665   27592.5340  subfind_density()/subfind_density.c/96
  59 2990 0             DM_Vz    126.4665   27719.0005  subfind_density()/subfind_density.c/97
  59 2991 0    	  DensDataGet    114.0667   27833.0673  subfind_density()/subfind_density.c/161
  68 2976 0   DomainNodeIndex      0.3498   19509.0337  force_treeallocate()/forcetree.c/4800
  68 2977 0        Nodes_base   3737.6087   23246.6424  force_treeallocate()/forcetree.c/4803
  68 2978 0     Extnodes_base   3114.6739   26361.3164  force_treeallocate()/forcetree.c/4811
  59 2992 0  	  DensDataResult     85.5500   27918.6173  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=59: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=81.3827 MB).
task 59: endrun called with an error level of 812


  68 2979 0          Nextnode    222.3054   26583.6217  force_treeallocate()/forcetree.c/4820
  68 2980 0            Father    221.9056   26805.5273  force_treeallocate()/forcetree.c/4827
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 59
  68 2981 0           Ngblist     63.4418   26868.9691  subfind_density()/subfind_density.c/78
  68 2982 0         Dist2list    126.8835   26995.8526  subfind_density()/subfind_density.c/79
  68 2983 0    DataIndexTable     21.8182   27017.6708  subfind_density()/subfind_density.c/87
  68 2984 0      DataNodeList     58.1818   27075.8526  subfind_density()/subfind_density.c/89
  68 2985 0              Left    126.8835   27202.7361  subfind_density()/subfind_density.c/91
  68 2986 0             Right    126.8835   27329.6196  subfind_density()/subfind_density.c/92
  68 2987 0              Todo     15.8604   27345.4800  subfind_density()/subfind_density.c/93
  68 2988 0             DM_Vx    126.8835   27472.3635  subfind_density()/subfind_density.c/95
  68 2989 0             DM_Vy    126.8835   27599.2470  subfind_density()/subfind_density.c/96
  68 2990 0             DM_Vz    126.8835   27726.1305  subfind_density()/subfind_density.c/97
  68 2991 0    	  DensDataGet    130.5704   27856.7010  subfind_density()/subfind_density.c/161
  68 2992 0  	  DensDataResult     97.9278   27954.6288  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=68: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=45.3712 MB).
task 68: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 68
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  50 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  50 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  50 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  50 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  50 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  50 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  50 2973 0         FOF_PList    380.6058   18861.4623  fof_fof()/fof.c/249
  50 2974 0         FOF_GList    634.3430   19495.8053  fof_fof()/fof.c/337
  50 2975 0             Group     11.0037   19506.8090  fof_fof()/fof.c/380
  50 2976 0   DomainNodeIndex      0.3498   19507.1588  force_treeallocate()/forcetree.c/4800
  50 2977 0        Nodes_base   3737.6087   23244.7675  force_treeallocate()/forcetree.c/4803
  50 2978 0     Extnodes_base   3114.6739   26359.4414  force_treeallocate()/forcetree.c/4811
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  50 2979 0          Nextnode    222.3054   26581.7468  force_treeallocate()/forcetree.c/4820
  50 2980 0            Father    221.9056   26803.6524  force_treeallocate()/forcetree.c/4827
  50 2981 0           Ngblist     63.4343   26867.0867  subfind_density()/subfind_density.c/78
  50 2982 0         Dist2list    126.8686   26993.9553  subfind_density()/subfind_density.c/79
  77 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  77 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  77 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  77 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  77 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  77 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  77 2973 0         FOF_PList    380.4184   18861.2749  fof_fof()/fof.c/249
  77 2974 0         FOF_GList    634.0306   19495.3055  fof_fof()/fof.c/337
  77 2975 0             Group     10.9197   19506.2252  fof_fof()/fof.c/380
  77 2976 0   DomainNodeIndex      0.3498   19506.5750  force_treeallocate()/forcetree.c/4800
  77 2977 0        Nodes_base   3737.6087   23244.1837  force_treeallocate()/forcetree.c/4803
  77 2978 0     Extnodes_base   3114.6739   26358.8576  force_treeallocate()/forcetree.c/4811
  77 2979 0          Nextnode    222.3054   26581.1630  force_treeallocate()/forcetree.c/4820
  50 2983 0    DataIndexTable     21.8182   27015.7735  subfind_density()/subfind_density.c/87
  50 2984 0      DataNodeList     58.1818   27073.9552  subfind_density()/subfind_density.c/89
  50 2985 0              Left    126.8686   27200.8238  subfind_density()/subfind_density.c/91
  77 2980 0            Father    221.9056   26803.0686  force_treeallocate()/forcetree.c/4827
  77 2981 0           Ngblist     63.4031   26866.4716  subfind_density()/subfind_density.c/78
  77 2982 0         Dist2list    126.8061   26993.2778  subfind_density()/subfind_density.c/79
  50 2986 0             Right    126.8686   27327.6924  subfind_density()/subfind_density.c/92
  50 2987 0              Todo     15.8586   27343.5510  subfind_density()/subfind_density.c/93
  50 2988 0             DM_Vx    126.8686   27470.4196  subfind_density()/subfind_density.c/95
  77 2983 0    DataIndexTable     21.8182   27015.0959  subfind_density()/subfind_density.c/87
  77 2984 0      DataNodeList     58.1818   27073.2777  subfind_density()/subfind_density.c/89
  77 2985 0              Left    126.8061   27200.0839  subfind_density()/subfind_density.c/91
  50 2989 0             DM_Vy    126.8686   27597.2882  subfind_density()/subfind_density.c/96
  50 2990 0             DM_Vz    126.8686   27724.1568  subfind_density()/subfind_density.c/97
  50 2991 0    	  DensDataGet    141.1854   27865.3421  subfind_density()/subfind_density.c/161
  77 2986 0             Right    126.8061   27326.8900  subfind_density()/subfind_density.c/92
  77 2987 0              Todo     15.8508   27342.7408  subfind_density()/subfind_density.c/93
  50 2992 0  	  DensDataResult    105.8890   27971.2312  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------
  77 2988 0             DM_Vx    126.8061   27469.5469  subfind_density()/subfind_density.c/95
  77 2989 0             DM_Vy    126.8061   27596.3530  subfind_density()/subfind_density.c/96
  77 2990 0             DM_Vz    126.8061   27723.1591  subfind_density()/subfind_density.c/97

Task=50: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=28.7688 MB).
task 50: endrun called with an error level of 812


  77 2991 0    	  DensDataGet    115.5226   27838.6818  subfind_density()/subfind_density.c/161
application called MPI_Abort(MPI_COMM_WORLD, 812) - process 50
  77 2992 0  	  DensDataResult     86.6420   27925.3237  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=77: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=74.6763 MB).
task 77: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 77
------------------------ Allocated Memory Blocks----------------------------------------
Task   Nr F          Variable      MBytes   Cumulative         Function/File/Linenumber
----------------------------------------------------------------------------------------
  51 2965 0     ProcessedFlag     55.4764      57.4332  allocate_memory()/allocate.c/52
  51 2966 0  NextActiveParticle    221.9056     279.3388  allocate_memory()/allocate.c/55
  51 2967 0     NextInTimeBin    221.9056     501.2444  allocate_memory()/allocate.c/58
  51 2968 0     PrevInTimeBin    221.9056     723.1499  allocate_memory()/allocate.c/61
  51 2969 0                 P  17752.4469   18475.5968  allocate_memory()/allocate.c/67
  51 2972 0          TopNodes      5.1972   18480.8565  domain_allocate()/domain.c/570
  51 2973 0         FOF_PList    380.6110   18861.4675  fof_fof()/fof.c/249
  51 2974 0         FOF_GList    634.3516   19495.8191  fof_fof()/fof.c/337
  51 2975 0             Group     12.6255   19508.4446  fof_fof()/fof.c/380
  51 2976 0   DomainNodeIndex      0.3498   19508.7944  force_treeallocate()/forcetree.c/4800
  51 2977 0        Nodes_base   3737.6087   23246.4031  force_treeallocate()/forcetree.c/4803
  51 2978 0     Extnodes_base   3114.6739   26361.0770  force_treeallocate()/forcetree.c/4811
  51 2979 0          Nextnode    222.3054   26583.3824  force_treeallocate()/forcetree.c/4820
  51 2980 0            Father    221.9056   26805.2880  force_treeallocate()/forcetree.c/4827
  51 2981 0           Ngblist     63.4352   26868.7232  subfind_density()/subfind_density.c/78
  51 2982 0         Dist2list    126.8703   26995.5935  subfind_density()/subfind_density.c/79
  51 2983 0    DataIndexTable     21.8182   27017.4117  subfind_density()/subfind_density.c/87
  51 2984 0      DataNodeList     58.1818   27075.5935  subfind_density()/subfind_density.c/89
  51 2985 0              Left    126.8703   27202.4638  subfind_density()/subfind_density.c/91
  51 2986 0             Right    126.8703   27329.3341  subfind_density()/subfind_density.c/92
  51 2987 0              Todo     15.8588   27345.1929  subfind_density()/subfind_density.c/93
  51 2988 0             DM_Vx    126.8703   27472.0632  subfind_density()/subfind_density.c/95
  51 2989 0             DM_Vy    126.8703   27598.9336  subfind_density()/subfind_density.c/96
  51 2990 0             DM_Vz    126.8703   27725.8039  subfind_density()/subfind_density.c/97
  51 2991 0    	  DensDataGet    124.2881   27850.0920  subfind_density()/subfind_density.c/161
  51 2992 0  	  DensDataResult     93.2161   27943.3080  subfind_density()/subfind_density.c/201
----------------------------------------------------------------------------------------

Task=51: Not enough memory in mymalloc_fullinfo() to allocate 87.2727 MB for variable '	  DensDataOut' at subfind_density()/subfind_density.c/line 203 (FreeBytes=56.692 MB).
task 51: endrun called with an error level of 812


application called MPI_Abort(MPI_COMM_WORLD, 812) - process 51
