If you are using lightgbm or similar packages, I would suggest you to leave them as it is, they can handle NaN values by default, quite well. Another tip could be using a very large negative number say -99999, so that its handled it completely different from other values. You can also try to polate the values. Mean, Median Mode imputation, I would not suggest in this case.