Looking at it from outside as an emotionally detached third party, it is clear to me that (sad but true) the record collection belongs to your friend, since you didn't purchase it from him and he obviously does not remember giving it to you as a gift, and still considers you its caretaker, which is why
now he says he wants to come to my house and pack everything up personally and anything else I may have got during the move like wall pictures, art, desk, etc.
The fact that the records belong to him puts you at a distinct disadvantage in this matter, although your friend was extremely lucky that you could help him retain the collection.
So approach him gracefully says I, not to persuade him to give up ownership of the collection, but to convince him to give you continued access to the records in the same way that you stored them all these years and enabled him to access them whenever he wanted to.
If you are more concerned about permanently saving the musical material in that collection (which is what I would be concerned about, seeing that the discs don't belong to me), especially if you think he is likely to sell the discs without consulting with you in future, then you can consider asking your friend to allow you to record everything or only selected discs, either ason vinyl fiscs or or as digital files, which is your moral right for having saved his collection when he could have lost it all. This is a compromise solution that your friend might readily agree to, as long as you are willing to not dispute the ownership of the records.
A calm, friendly and neutrally-toned discussion with your friend will help you to understand his views and convey yoursyour points without any emotional accusations from either side, so that the two of you can remain friends and work out a mutually satisfactory solution.