What I'm talking about is taking the other person's part private key and generating a vanity wallet address
Don't you mean
public key?
No.
Part private key is what I said. Feel free to read up on the subject then we can pick up this conversation where I'm now going to leave it off (until you read up on the subject that is).
I can't see any reason why the other party needs "private" key, partial or not. They only need the public key and then they can move from there by incrementing that point one G at a time until they find the correct public key that generates the desired address. Then all they have to do is to send back the number of times they added G to that point. User can simply add that value to their private key and get the new private key which corresponds to the public key of the vanity address.
It doesn't work like that - as I said, I encourage you to read up on the subject.