If someone steals your physical hardware wallet, it doesn’t necessarily mean that they have access to your coins.They would also need to know your PIN or password that you created when you first set up the device. (Ledger devices ask for you to create a 4-7 digit PIN) Also as an additional layer of security Ledger devices also only allow for a very limited amount of attempts for inputting your PIN, so if someone like a thief is trying to guess, most likely they will be locked out.
Your seed phrase represents the private key to your hardware wallet. It is that set of 12 , 18 or 24 words that you were prompted to write down and keep in a safe place away from where you store your hardware wallet.
Pro tips: Store your seed phrase on a secure medium (not saved to your notes on your computer desktop or in your email and also in a secure location that is as safe as possible from exposure to things like water, fire, and the attention of people wanting to steal from you.
Seriously consider having a replacement device already on hand in the event that your current device is lost, stolen or damaged. This will make the recovery time much faster and you won’t need to wait for the new hardware wallet to be dispatched and shipped to you (during a bull run when a lot more people enter this space, things like hardware wallet manufactures often become backlogged and you will most likely have a long wait time before receiving the new device.