Do you define the validity of a state transition with Schema?

Yes. You always validate the state transition against the Schema rules. A state transition is valid only if it follows and matches the Schema rules. Thus, RGB smart contract is in a valid state if all state transitions which you know about are in a valid state.

To compare, in Ethereum the state of a given smart contract is valid if it is mined on the blockchain, and the state change of a smart contract should also be mined on the blockchain. Thus, Ethereum smart contract has a defined state known to the whole world and valid by definition because it's a part of the blockchain history. In order to update that state you need to run a specific algorithm defined and implemented in the smart contract code at the moment of its creation. It makes the options of what contract can and cannot do very restricted.

RGB works differently. With RGB you define the evolution rules at the level of Schema, but you don't restrict the future operations of the whole smart contract by introducing a global algorithm to which every state should comply. You have only local operations performed by every node and both the state and the smart contract itself are valid until each of those historical operations matches the Schema-defined rules (but again, you're not restricting all the possibilities of evolution of the history of contract with a predefined algorithm). Basically, a contract can behave very differently, until each of the changes matches certain validation rules. So it's more like local rules instead of global algorithms. Ethereum has global algorithm and you always operate the whole state of the smart contract. In RGB you operate a local portion of the state with locally applied rules, which gives you much richer range of options.