MonetDB COPY INTO table "unexpected end of file"

Question

I am loading a sizeable csv file (300GB) into a tables using COPY INTO statement. After a long waiting time, I am getting an "unexpected end of file" exception and the table if empty after that.

Heres's my import query:

COPY d1 FROM '/home/d1_data/d1.csv'

And my csv data file:

2019-02-01T00:00:10,st0,0.839071,0.179288,0.585304,0.679371,0.492911,0.056175,0.498442,0.938126,0.668068,0.929086,0.081897,0.843644,0.974037,0.159324,0.142218,0.140207,0.625254,0.425917,0.771387,0.096174,0.120735,0.725770,0.139911,0.310633,0.382543,0.896953,0.445951,0.119868,0.424562,0.181185,0.379519,0.105958,0.845021,0.533097,0.723558,0.944910,0.036968,0.112205,0.799767,0.728473,0.968308,0.111421,0.905472,0.980631,0.865119,0.293025,0.973192,0.408123,0.272021,0.125133,0.763793,0.819480,0.600016,0.178615,0.777532,0.081147,0.652687,0.458067,0.767267,0.711449,0.957630,0.115871,0.569370,0.517578,0.093003,0.682874,0.679829,0.485540,0.926170,0.080369,0.570393,0.484541,0.568747,0.626574,0.117149,0.715187,0.655418,0.276893,0.841691,0.173985,0.805234,0.241210,0.858166,0.021120,0.224665,0.238334,0.864353,0.103404,0.868038,0.992483,0.624129,0.755107,0.620674,0.763600,0.199850,0.396798,0.612075,0.515486,0.961466,0.434988
2019-02-01T00:00:20,st0,0.322934,0.755268,0.061692,0.212437,0.231739,0.826009,0.402892,0.546866,0.748315,0.428897,0.634761,0.384299,0.192479,0.391302,0.920955,0.526497,0.150713,0.338057,0.933859,0.137499,0.875741,0.228530,0.297205,0.266878,0.288009,0.060985,0.882594,0.490286,0.870628,0.317989,0.476885,0.132587,0.459073,0.457800,0.380606,0.978631,0.687570,0.353860,0.224363,0.931935,0.272906,0.443753,0.908269,0.173270,0.567581,0.705271,0.659782,0.530196,0.615158,0.107020,0.337759,0.287402,0.113100,0.750601,0.380647,0.338062,0.470644,0.560054,0.916784,0.102615,0.653475,0.234832,0.241591,0.092253,0.984721,0.061122,0.418502,0.268967,0.170532,0.623880,0.505132,0.659034,0.752930,0.888594,0.871888,0.676820,0.938585,0.050625,0.063221,0.559219,0.451311,0.844238,0.915815,0.935894,0.918915,0.271461,0.099396,0.661230,0.405390,0.608056,0.919490,0.483303,0.240281,0.329818,0.181569,0.511471,0.432861,0.463347,0.560382,0.855283
2019-02-01T00:00:30,st0,0.692054,0.538778,0.764992,0.656943,0.006166,0.610429,0.479586,0.639454,0.107885,0.338176,0.535457,0.871265,0.291767,0.955159,0.271295,0.421824,0.772407,0.531340,0.419594,0.776071,0.452270,0.281994,0.479907,0.745093,0.627713,0.774344,0.699013,0.587567,0.878019,0.153955,0.986209,0.704153,0.783832,0.704486,0.200587,0.630304,0.235955,0.429266,0.752330,0.484207,0.394956,0.518921,0.688756,0.720469,0.056679,0.160093,0.502845,0.915870,0.359901,0.744948,0.005774,0.194809,0.180417,0.100580,0.428749,0.621978,0.782535,0.834345,0.960411,0.703126,0.681373,0.894144,0.943699,0.037323,0.294162,0.047351,0.940178,0.396505,0.243780,0.410479,0.257793,0.581372,0.235662,0.441054,0.536284,0.588570,0.946028,0.466676,0.124679,0.803133,0.713820,0.810444,0.810953,0.259700,0.450738,0.995637,0.339662,0.132606,0.189827,0.208749,0.430025,0.843661,0.706039,0.650623,0.797073,0.719763,0.055521,0.852340,0.396091,0.429506

Any idea on why is this exception happening?

Answer 1

The "unexpected end of file" error is issued by mclient when the connection to the server is unexpectedly broken due to an overhead. The issue was that I was running too many loads at the same time.

My machine has a 32 cores and 128GB RAM on Ubuntu 20.04 so I thought I am safe wrt to the hardware ressources but that wasn't the case. In fact, the overhead has cause mclient to crash.

To solve/avoid this problem, I would recommend keeping the task overhead wrt to the machines characteristics in mind, as well as running heavy tasks sequentially as opposed to parallelly.