Primary competition visual

RMB Monthly Mini Challenge #3: Nowcast June

Helping South Africa
$600 USD
Challenge completed over 2 years ago
Forecast
259 joined
85 active
Starti
Jun 01, 23
Closei
Jul 01, 23
Reveali
Jul 21, 23
User avatar
skaak
Ferra Solutions
Latest result - bit of a surprise
Data · 19 Jul 2023, 08:12 · 33

Wow, what a surprise on some of the components ... here the data so long

ID Value
June_food and non-alcoholic beverages 118.3
June_alcoholic beverages and tobacco 110.9
June_clothing and footwear 104.3
June_housing and utilities 105.4
June_household contents and services 107.7
June_health 110.8
June_transport 112.3
June_communication 99.6
June_recreation and culture 105.3
June_education 110.4
June_restaurants and hotels 110.0
June_miscellaneous goods and services 109.6
June_headline CPI 109.8
Discussion 33 answers
User avatar
Koleshjr
Multimedia university of kenya

A shift from 0.4785 lb to 1.05..... actual, crazy

19 Jul 2023, 08:52
Upvotes 0
User avatar
skaak
Ferra Solutions

Yes, this will shuffle the LB quite a bit ... are you going to print 1.05 on private? At least (take this as a *huge* compliment) I might have some small thing to celebrate about ... :-)

User avatar
Koleshjr
Multimedia university of kenya

😂😂yes , I'm going to print 1.05 on private but we had better ones which never performed well on public with much lower scores, (0.68)

User avatar
skaak
Ferra Solutions

Well, fwiw I think you have a really good & strong model. Problem (we all) now have is what to do with this? How much do we adjust to this or should we ignore it ...

User avatar
Koleshjr
Multimedia university of kenya

That's the big question ..anyways see you in July's :)

User avatar
Yisakberhanu
wachemo university

i think because of transport and hotels cpi value

RMSE of 0.71 here

19 Jul 2023, 10:40
Upvotes 0
User avatar
skaak
Ferra Solutions

Nice jump - congrats.

User avatar
Koleshjr
Multimedia university of kenya

👏👏

I thought it was terrible

User avatar
ff
University of Yaoundé I

"Transport" was difficult to predict. difference of almost 2 points

19 Jul 2023, 12:56
Upvotes 2
User avatar
skaak
Ferra Solutions

And food was a big surprise to me ... sugar grew 5+% from May! I was hoping food prices would come down a bit.

User avatar
Jaw22
Zindi africa

The dance of cpi or is it a tango...have to build in sentiment and cpi news and commentating perhaps.

19 Jul 2023, 14:11
Upvotes 1
User avatar
skaak
Ferra Solutions

Wow @Jaw22 if you do all that you must sell the model for big bucs. Remember you must let it run unattended for three months in the big competition so what you describe will be very tricky I think. Even just to get transport right would require a deep petrol price markup type model I think that will be difficult to automate entirely. But you've been doing really well in the CPIs lately - can't wait to see your position on the private LB.

User avatar
Jaw22
Zindi africa

Dankie Skaak, I need to block out time to work on my models again. Kind Regards

User avatar
Kamal_Moha

RMSE of 0.5 here which is awesome

19 Jul 2023, 20:15
Upvotes 0
User avatar
skaak
Ferra Solutions

Hmmm yes that is good and similar to public, I wish zindi would publish the private now.

User avatar
Yisakberhanu
wachemo university

since we have no enough predictors data, special for transport , hotels, and food, it will be impossible to forcast the cpi value. Also validation is so so so so so difficult.

20 Jul 2023, 06:39
Upvotes 0
User avatar
skaak
Ferra Solutions

Don't overthink it ... simply fit model in rolling fashion and forecast t+1 for last few months?

User avatar
Yisakberhanu
wachemo university

I got you

User avatar
enigmatic

rmse of about 0.54, i guess i was overfitting the lb, i had some values with rmse as low as 0.32. <lesson>have confidence in your private result</lesson learnt>

20 Jul 2023, 12:55
Upvotes 0
User avatar
skaak
Ferra Solutions

My goodness @enigmatic, I think at 0.54 you are going to be very well placed on private LB, but I wonder what RMB (public LB truth) would score here.

Come on @zindi we are waiting anxiously for release of private.

User avatar
enigmatic

Can you imagine. I was calculating mse. I'll just wait for zindi's result

User avatar
ff
University of Yaoundé I

😄😄😄😄

sqrt(0.54) = 0.7348

User avatar
enigmatic

Lol, it is what it is.

User avatar
Amy_Bray
Zindi

T - 15 minutes until the private leaderboard is revealed, talk about a shake up!

21 Jul 2023, 06:44
Upvotes 0
User avatar
tomwetherell
Met Office

Ended up with 0.420548253. Very happily surprised.

I actually forgot to select my new model as a submission, which is a little frustrating as it did better than my old one. Oh well - definitely not upset with what I ended up with.

22 Jul 2023, 12:21
Upvotes 0
User avatar
skaak
Ferra Solutions

Wow! Congrats, you are really doing well in the CPI minis, and to go in with such a public LB and (presumably) not worry.

EDIT: ...

Just three subs - so it seems you just rerun your previous models, which must be quite good. So if you selected a different 2/3 models you'd be even better. Would you also be in a better LB position? Or perhaps better is to ask what do you mean old and new model?

I undertook to just play the same two models all the time, but made few small changes in most recent comp ... good thing too, as my "new" model did a bit better. This is tricky, I don't want to overplay it, just cook up a quick model and stick to it throughout, but then it is so nice to tweak a model.

User avatar
tomwetherell
Met Office

Thanks! I was a little worried when I saw the public leaderboard score to be honest, but I was submitting on the last day so I had no choice but to trust my predictions.

By old model I mean the model I developed for the first competition. Over the last month and a bit I've been working on a new model which involves a lot more features than the first. The new model did better in June than the old one, but I forgot to select it. It got a 0.3266 private score.

I properly tested the new model last week and it turns out it's actually weaker than the old model, in all except 2 of the categories (must have just got lucky in June). So I definitely agree it's probably best to stick with something simple instead of tweaking ... but I enjoy tweaking so I probably won't stop :)

User avatar
skaak
Ferra Solutions

I see you just subbed - for July nobody has good LB, seems we are not agreeing with RMB's prediction.

I've been gradually complicating my model over time ... did not test it really, but it is not just complications for the sake of them, rather plugging gaps in the original model, so I think (hope!) each one is incrementally stronger. Initially I had a really really really simple model, almost just to proof a point, but I knew it would expire quickly. In each round, I more or less subbed one simple and one more complex model and the complex one consistently outperformed, so next month it becomes the simple and I tweak it a bit into the new complex.

The CPI landscape is anyhow so fluid ... so I think ahead a bit also, what will work from now until the end of the year. Easier said than done.

Oh well - hope you have another good round in July! See you there.

User avatar
Jaw22
Zindi africa

I am following you on the LB, been experimenting and tweaking here and there...I am a bit pressed for time. Exciting though, interesting to see what happen in the finals. All the best.

User avatar
skaak
Ferra Solutions

@Jaw22 !!! You are doing so well in these - I am impressed. I noted you in July, seems we are in agreement, at least in how far off RMB could be :-).

This is "for real" so even a simple model can outperform a good one, but I think in the finals, over three months, too simple (or too complex) will trip you up and you have to be careful.

Every time I say now I am done and then in the next mini I end up tweaking a bit. All the best to you also my friend.

User avatar
Jaw22
Zindi africa

@Skaak, thank you for the vote of confidence skaak, but the first three places on the LB remain elusive look like a Gremlin in the works...