An old saying that Dads are likely to have heard is “an elephant never forgets”. So “I have the memory of an elephant” is saying “I never forget.”

Unless you’re making a Dad joke.

In which case it means “I remember seeing an elephant once”.

There is an idea or saying that elephants never forget or that they have very good memories First article in google about this.

So to say you have a memory of an elephant could mean your memory is very good.

The twist is that this can also mean you remember seeing an elephant, which is what the second line implies.