How to disable the file removal, when the file is in upload stage or in processing

I want to disable the file deletion until the file is in upload stage or in processing stage. Only the remove icon (removal feature) will get activated once the file is successfully uploaded or upload-failed.