They can refuse if the loan outstanding is much more than the collateral provided.
Ex: If you have a loan outstanding of 100,000$ and you have provided a collateral of 50,000$ you cannot expect the bank to release any collateral.
Lets say your outstanding is only $30,000 then you can expect the bank to release a certain portion of the collateral atleast $20,000