tag:blogger.com,1999:blog-6141980.post455581453108062083..comments2024-03-20T12:28:35.004-05:00Comments on Nuit Blanche: That Netflix RMSE is way too low or is it ? ( Clustering-Based Matrix Factorization - implementation -)Igorhttp://www.blogger.com/profile/17474880327699002140noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-6141980.post-39011384892792530482013-02-11T11:26:26.390-06:002013-02-11T11:26:26.390-06:00Zeno,
Yes, this is my understanding. The RMSE doe...Zeno,<br /><br />Yes, this is my understanding. The RMSE does not seem to hold.<br /><br />IgorIgorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-24917477269166642062013-02-11T09:26:45.436-06:002013-02-11T09:26:45.436-06:00Hi Igor,
the bug caused the RMSE on MovieLens to ...Hi Igor,<br /><br />the bug caused the RMSE on MovieLens to be vastly underestimated, so I guess that the results on Netflix do not hold any more.Zenohttps://www.blogger.com/profile/01719463815974213213noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-66801969502101788082013-02-10T08:55:06.275-06:002013-02-10T08:55:06.275-06:00Irchans,
If you followed the discussion on the li...Irchans,<br /><br />If you followed the discussion on the linkedin group on advanced matrix factorization, you would have noticed that, most probably Zeno helped a lot. Right now, I am personally waiting for Nima to confirm if the bug is substantial or merely changes the results (while still beating the netflix RMSE).<br /><br /><br />Cheers,<br /><br />Igor.Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-54307813767205464442013-02-10T07:51:17.059-06:002013-02-10T07:51:17.059-06:00Nima,
Did you find the mistake or did one of th...Nima,<br /> Did you find the mistake or did one of the Nuit Blanche readers find the mistake? Just curious. <br /><br />PS: Nice Paperirchanshttps://www.blogger.com/profile/10845563978677248090noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-32069615196318006232013-02-08T18:03:51.476-06:002013-02-08T18:03:51.476-06:00Just want to update you guys that the results were...Just want to update you guys that the results were not valid. I had a mistake in my code. I will update the paper with new results soon.Nimanoreply@blogger.comtag:blogger.com,1999:blog-6141980.post-46456056256657853392013-02-08T08:24:47.922-06:002013-02-08T08:24:47.922-06:00@Nima The measure is not the only part of the prot...@Nima The measure is not the only part of the protocol.Zenohttps://www.blogger.com/profile/01719463815974213213noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-77510729742677112772013-02-08T07:59:19.494-06:002013-02-08T07:59:19.494-06:00Hello everyone, I am the author.. Before starting ...Hello everyone, I am the author.. Before starting to answer your comments I gonna ask you not to decide so fast before reading the paper..<br /><br />@winsty i am using ratings 4 or above 4 only for clustering purpose and it doesn't make any change in train set or test set and I am pretty sure that we don't miss any of them in our evaluation. Even users or items with that their all ratings are under 4 will go to same clusters... I have made the Netflix dataset that I used online....<br /><br />@igor as I say in the paper if you don't use the threshold for stoping the learning process the MF model will get in an overfitting. After 100 epoches I've got RMSE .90 for basic matrix factorization and using the threshold it is almost .81...<br /><br />@zeno I think they were using RMSE? They were not?Nimahttp://www.linkedin.com/groupItem?view&srchtype=discussedNews&gid=4084620&item=211708878&type=member&trk=eml-anet_dig-b_pd-ttl-cn&ut=0km90OlCMkYRA1&_mSplash=1noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-39716739092571388872013-02-08T04:47:49.218-06:002013-02-08T04:47:49.218-06:00winsty,
"...They just kept all the items wi...winsty, <br /><br />"...They just kept all the items with at least 4 ratings... The results are totally not comparable..." is a good observation. However " To my best of my knowledge, on Movielens 100K dataset, the best single model could only reach an RMSE about 0.88. In their paper, the basic MF model could even reach 0.81..." is not really helpful. The reason the code is shared is for people to explain **why** we seem to be getting extraordinarly better results. We are not going through a literature review process.Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-91039208914115177522013-02-08T04:33:46.238-06:002013-02-08T04:33:46.238-06:00They just kept all the items with at least 4 ratin...They just kept all the items with at least 4 ratings... The results are totally not comparable. To my best of my knowledge, on Movielens 100K dataset, the best single model could only reach an RMSE about 0.88. In their paper, the basic MF model could even reach 0.81...winstyhttps://www.blogger.com/profile/12181770590663366493noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-13547436828190784062013-02-07T10:21:37.723-06:002013-02-07T10:21:37.723-06:00Zeno,
you may want to give your inoput directly t...Zeno,<br /><br />you may want to give your inoput directly to Nima in the Linkedin thread:<br /><br />http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&discussionID=211708878&gid=4084620&commentID=118330837&trk=view_disc&ut=2YFxQFkHY3XBA1<br /><br />Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-10649726811525792452013-02-07T10:10:12.693-06:002013-02-07T10:10:12.693-06:00According to the paper, they do not use the same e...According to the paper, they do not use the same evaluation protocol as the one used in the Netflix prize competition.<br /><br />So the results are not comparable.Zenohttps://www.blogger.com/profile/01719463815974213213noreply@blogger.com