Generally, you'll need an acid for this. I strongly suggest that you test ANY chems in an inconspicuous area. I would start with a product from Northern Tool called BRIXX WASH, which is the equivalent of a much more expensive Prosoco chem. Use it full strength. ANY chem you use may take multiple applications, but BRIXX is buffered and less likely to harm the brick or mortar. Upon application, the affected area should bubble (Please wear appropriate PPE) When the bubbling (the reaction of the acid and minerals) stops, reapply if there is still residue. Some pressure MAY be desired to "coax" it off.
As a last resort, muriatic acid may be the way to go, but be aware that it is possible to "burn" the brick/mortar, turning it brown. Follow directions completely. Wet brick, apply and let foam, rinse immediately and repeat until its gone. Again PPE is extremely important since the fumes can really do a number on your lungs.
As far as the plants, if they are right against the brick, it may not be possible to prevent damage. If there is any distance, wet them and the sail thoroughly and you should have no problems. Temporary removal of the plants may be necessary.
Working with acids can produce very satisfying results, but the consequences of "screwing it up" can be dangerous and expensive. Test, Test, Test. If possible, post a pic or two so that you can have a few more pairs of eyes see the problem and possibly offer better advice.
