Cuda desteği olmadan sadece işlemci ile Mac’de GLASS çalıştırmak çılgınlık olabilir. Çözmek zorunda olduğunuz onlarca parametre ayarı dışında saatlerce beklemek ve beklemelerin hatalarla kesiliyor olması tam bir işkence olabilir.
İşlemin sonunda gözlerinizi yaşartacak olan sonucu paylaşıyorum. Maciniz CUDA yerine kendi grafik işlemcisini Python ile Görsel algılama Anomali detection’da kullanıyor.
Apple M1 Max işlemcide GPU (Metal Performance Shaders – MPS) desteğiyle çalıştırmak için GLASS projesini uyarlamak, CUDA yerine MPS backend’ini kullanmayı gerektirir. Aşağıda, bu işlemi adım adım açıklıyorum:
1. Proje Hazırlığı
GLASS projesini ve bağımlılıkları kurmak için aşağıdaki adımları izleyin:
Projenin Klonlanması
git clone https://github.com/cqylunlun/glass.git
cd glass
Python Ortamının Hazırlanması
Yeni bir Python ortamı oluşturun ve gerekli paketleri yükleyin:
conda create -n glass_mps python=3.9
conda activate glass_mps
pip install -r requirements.txt
PyTorch’un MPS desteği için özel bir yükleme yapın:
pip install torch torchvision torchaudio –index-url https://download.pytorch.org/whl/cpu
2. CUDA Yerine MPS Backend’ini Kullanma
CUDA yerine Metal (MPS) backend’ini kullanmak için proje kodunda device seçimi yapmanız gerekiyor.
device Ayarlarını Değiştirin
GLASS projesindeki PyTorch modelinin çalıştığı cihazı seçen kısımları bulun. Bunlar genellikle torch.device(“cuda”) olarak tanımlıdır. Bu satırları aşağıdaki gibi değiştirin:
Örnek:
main.py, train.py veya utils.py dosyalarında şu değişiklikleri yapın:
import torch
# Varsayılan cihaz seçimi
device = torch.device(“mps” if torch.backends.mps.is_available() else “cpu”)
print(f”Using device: {device}”)
Benim main.py içeriğim aşağıdaki gibi oldu:
from datetime import datetime
import pandas as pd
import os
import logging
import sys
import click
import torch
import warnings
import backbones
import glass
import utilslogging.basicConfig(level=logging.INFO)
LOGGER = logging.getLogger(name)@click.group(chain=True)
@click.option(“–results_path”, type=str, default=”results”)
@click.option(“–gpu”, type=int, default=[0], multiple=True, show_default=True)
@click.option(“–seed”, type=int, default=0, show_default=True)
@click.option(“–log_group”, type=str, default=”group”)
@click.option(“–log_project”, type=str, default=”project”)
@click.option(“–run_name”, type=str, default=”test”)
@click.option(“–test”, type=str, default=”ckpt”)
def main(**kwargs):
pass@main.command(“net”)
@click.option(“–dsc_margin”, type=float, default=0.5)
@click.option(“–train_backbone”, is_flag=True)
@click.option(“–backbone_names”, “-b”, type=str, multiple=True, default=[])
@click.option(“–layers_to_extract_from”, “-le”, type=str, multiple=True, default=[])
@click.option(“–pretrain_embed_dimension”, type=int, default=1024)
@click.option(“–target_embed_dimension”, type=int, default=1024)
@click.option(“–patchsize”, type=int, default=3)
@click.option(“–meta_epochs”, type=int, default=640)
@click.option(“–eval_epochs”, type=int, default=1)
@click.option(“–dsc_layers”, type=int, default=2)
@click.option(“–dsc_hidden”, type=int, default=1024)
@click.option(“–pre_proj”, type=int, default=1)
@click.option(“–mining”, type=int, default=1)
@click.option(“–noise”, type=float, default=0.015)
@click.option(“–radius”, type=float, default=0.75)
@click.option(“–p”, type=float, default=0.5)
@click.option(“–lr”, type=float, default=0.0001)
@click.option(“–svd”, type=int, default=0)
@click.option(“–step”, type=int, default=20)
@click.option(“–limit”, type=int, default=392)
def net(
backbone_names,
layers_to_extract_from,
pretrain_embed_dimension,
target_embed_dimension,
patchsize,
meta_epochs,
eval_epochs,
dsc_layers,
dsc_hidden,
dsc_margin,
train_backbone,
pre_proj,
mining,
noise,
radius,
p,
lr,
svd,
step,
limit,
):
backbone_names = list(backbone_names)
if len(backbone_names) > 1:
layers_to_extract_from_coll = []
for idx in range(len(backbone_names)):
layers_to_extract_from_coll.append(layers_to_extract_from)
else:
layers_to_extract_from_coll = [layers_to_extract_from]def get_glass(input_shape, device): glasses = [] for backbone_name, layers_to_extract_from in zip(backbone_names, layers_to_extract_from_coll): backbone_seed = None if ".seed-" in backbone_name: backbone_name, backbone_seed = backbone_name.split(".seed-")[0], int(backbone_name.split("-")[-1]) backbone = backbones.load(backbone_name) backbone.name, backbone.seed = backbone_name, backbone_seed glass_inst = glass.GLASS(device) glass_inst.load( backbone=backbone, layers_to_extract_from=layers_to_extract_from, device=device, input_shape=input_shape, pretrain_embed_dimension=pretrain_embed_dimension, target_embed_dimension=target_embed_dimension, patchsize=patchsize, meta_epochs=meta_epochs, eval_epochs=eval_epochs, dsc_layers=dsc_layers, dsc_hidden=dsc_hidden, dsc_margin=dsc_margin, train_backbone=train_backbone, pre_proj=pre_proj, mining=mining, noise=noise, radius=radius, p=p, lr=lr, svd=svd, step=step, limit=limit, ) glasses.append(glass_inst.to(device)) return glasses return "get_glass", get_glass@main.command(“dataset”)
@click.argument(“name”, type=str)
@click.argument(“data_path”, type=click.Path(exists=True, file_okay=False))
@click.argument(“aug_path”, type=click.Path(exists=True, file_okay=False))
@click.option(“–subdatasets”, “-d”, multiple=True, type=str, required=True)
@click.option(“–batch_size”, default=8, type=int, show_default=True)
@click.option(“–num_workers”, default=16, type=int, show_default=True)
@click.option(“–resize”, default=288, type=int, show_default=True)
@click.option(“–imagesize”, default=288, type=int, show_default=True)
@click.option(“–rotate_degrees”, default=0, type=int)
@click.option(“–translate”, default=0, type=float)
@click.option(“–scale”, default=0.0, type=float)
@click.option(“–brightness”, default=0.0, type=float)
@click.option(“–contrast”, default=0.0, type=float)
@click.option(“–saturation”, default=0.0, type=float)
@click.option(“–gray”, default=0.0, type=float)
@click.option(“–hflip”, default=0.0, type=float)
@click.option(“–vflip”, default=0.0, type=float)
@click.option(“–distribution”, default=0, type=int)
@click.option(“–mean”, default=0.5, type=float)
@click.option(“–std”, default=0.1, type=float)
@click.option(“–fg”, default=1, type=int)
@click.option(“–rand_aug”, default=1, type=int)
@click.option(“–augment”, is_flag=True)
def dataset(
name,
data_path,
aug_path,
subdatasets,
batch_size,
resize,
imagesize,
num_workers,
rotate_degrees,
translate,
scale,
brightness,
contrast,
saturation,
gray,
hflip,
vflip,
distribution,
mean,
std,
fg,
rand_aug,
augment,
):
_DATASETS = {“mvtec”: [“datasets.mvtec”, “MVTecDataset”], “visa”: [“datasets.visa”, “VisADataset”],
“mpdd”: [“datasets.mvtec”, “MVTecDataset”], “wfdd”: [“datasets.mvtec”, “MVTecDataset”], }
dataset_info = _DATASETS[name]
dataset_library = import(dataset_info[0], fromlist=[dataset_info[1]])def get_dataloaders(seed, test, get_name=name): dataloaders = [] for subdataset in subdatasets: test_dataset = dataset_library.__dict__[dataset_info[1]]( data_path, aug_path, classname=subdataset, resize=resize, imagesize=imagesize, split=dataset_library.DatasetSplit.TEST, seed=seed, ) test_dataloader = torch.utils.data.DataLoader( test_dataset, batch_size=batch_size, shuffle=False, num_workers=num_workers, prefetch_factor=2, pin_memory=True, ) test_dataloader.name = get_name + "_" + subdataset if test == 'ckpt': train_dataset = dataset_library.__dict__[dataset_info[1]]( data_path, aug_path, dataset_name=get_name, classname=subdataset, resize=resize, imagesize=imagesize, split=dataset_library.DatasetSplit.TRAIN, seed=seed, rotate_degrees=rotate_degrees, translate=translate, brightness_factor=brightness, contrast_factor=contrast, saturation_factor=saturation, gray_p=gray, h_flip_p=hflip, v_flip_p=vflip, scale=scale, distribution=distribution, mean=mean, std=std, fg=fg, rand_aug=rand_aug, augment=augment, batch_size=batch_size, ) train_dataloader = torch.utils.data.DataLoader( train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_workers, prefetch_factor=2, pin_memory=True, ) train_dataloader.name = test_dataloader.name LOGGER.info(f"Dataset {subdataset.upper():^20}: train={len(train_dataset)} test={len(test_dataset)}") else: train_dataloader = test_dataloader LOGGER.info(f"Dataset {subdataset.upper():^20}: train={0} test={len(test_dataset)}") dataloader_dict = { "training": train_dataloader, "testing": test_dataloader, } dataloaders.append(dataloader_dict) print("\n") return dataloaders return "get_dataloaders", get_dataloaders@main.result_callback()
def run(
methods,
results_path,
gpu,
seed,
log_group,
log_project,
run_name,
test,
):
global LOGGER # Eklendi
methods = {key: item for (key, item) in methods}run_save_path = utils.create_storage_folder( results_path, log_project, log_group, run_name, mode="overwrite" ) list_of_dataloaders = methods["get_dataloaders"](seed, test) device = torch.device("mps" if torch.backends.mps.is_available() else "cpu") result_collect = [] data = {'Class': [], 'Distribution': [], 'Foreground': []} df = pd.DataFrame(data) for dataloader_count, dataloaders in enumerate(list_of_dataloaders): utils.fix_seeds(seed, device) dataset_name = dataloaders["training"].name imagesize = dataloaders["training"].dataset.imagesize glass_list = methods["get_glass"](imagesize, device) LOGGER.info( "Selecting dataset [{}] ({}/{}) {}".format( dataset_name, dataloader_count + 1, len(list_of_dataloaders), datetime.now().strftime('%Y-%m-%d %H:%M:%S'), ) ) models_dir = os.path.join(run_save_path, "models") os.makedirs(models_dir, exist_ok=True) for i, GLASS in enumerate(glass_list): flag = 0., 0., 0., 0., 0., -1. if GLASS.backbone.seed is not None: utils.fix_seeds(GLASS.backbone.seed, device) GLASS.set_model_dir(os.path.join(models_dir, f"backbone_{i}"), dataset_name) if test == 'ckpt': flag = GLASS.trainer(dataloaders["training"], dataloaders["testing"], dataset_name) if type(flag) == int: row_dist = {'Class': dataloaders["training"].name, 'Distribution': flag, 'Foreground': flag} df = pd.concat([df, pd.DataFrame(row_dist, index=[0])]) if type(flag) != int: i_auroc, i_ap, p_auroc, p_ap, p_pro, epoch = GLASS.tester(dataloaders["testing"], dataset_name) result_collect.append( { "dataset_name": dataset_name, "image_auroc": i_auroc, "image_ap": i_ap, "pixel_auroc": p_auroc, "pixel_ap": p_ap, "pixel_pro": p_pro, "best_epoch": epoch, } ) if epoch > -1: for key, item in result_collect[-1].items(): if isinstance(item, str): continue elif isinstance(item, int): print(f"{key}:{item}") else: print(f"{key}:{round(item * 100, 2)} ", end="") # save results csv after each category print("\n") result_metric_names = list(result_collect[-1].keys())[1:] result_dataset_names = [results["dataset_name"] for results in result_collect] result_scores = [list(results.values())[1:] for results in result_collect] utils.compute_and_store_final_results( run_save_path, result_scores, result_metric_names, row_names=result_dataset_names, ) # save distribution judgment xlsx after all categories if len(df['Class']) != 0: os.makedirs('./datasets/excel', exist_ok=True) xlsx_path = './datasets/excel/' + dataset_name.split('_')[0] + '_distribution.xlsx' df.to_excel(xlsx_path, index=False)if name == “main“:
warnings.filterwarnings(‘ignore’)
logging.basicConfig(level=logging.INFO)
LOGGER = logging.getLogger(name)# Komut satırı argümanlarını logla LOGGER.info("Command line arguments: {}".format(" ".join(sys.argv))) # Metal backend kontrolü if torch.backends.mps.is_available(): LOGGER.info("Using Metal (MPS) backend.") else: LOGGER.warning("Metal (MPS) backend is not available. Falling back to CPU.") main()
Ayrıca utils.py içeriğim aşağıdaki gibi oldu (Mac için)
import numpy as np
import csv
import os
import cv2
import random
import torch
import shutilIMAGENET_MEAN = [0.485, 0.456, 0.406]
IMAGENET_STD = [0.229, 0.224, 0.225]def distribution_judge(img, name):
“””Judge the distribution of specific category.Args: img: [np.array] Image of the category to be judged. name: [str] Name of the category. """ img_ = cv2.resize(img, (289, 289)) img = cv2.cvtColor(img_, cv2.COLOR_BGR2GRAY) img = cv2.blur(img, (39, 39)) dft = cv2.dft(np.float32(img), flags=cv2.DFT_COMPLEX_OUTPUT) dft_shift = np.fft.fftshift(dft) magnitude = 20 * np.log(cv2.magnitude(dft_shift[:, :, 0], dft_shift[:, :, 1])) magnitude[magnitude > 170] = 255 magnitude[magnitude <= 170] = 0 height, width = magnitude.shape center = (height // 2, width // 2) y_indices, x_indices = np.where(magnitude == 255) y_all, x_all = np.indices((2 * height, 2 * width)) l1_dist_x = np.abs(x_indices - center[1]) l1_dist_y = np.abs(y_indices - center[0]) dist = np.sqrt((x_indices - center[1]) ** 2 + (y_indices - center[0]) ** 2) l2_dist_all = np.sqrt((x_all - center[1]) ** 2 + (y_all - center[0]) ** 2) side_x = np.max(l1_dist_x) side_y = np.max(l1_dist_y) radius = np.max(dist) points_num = len(dist) l1_density = points_num / (4 * np.max([side_x, 1]) * np.max([side_y, 1])) l2_density = points_num / (np.sum(l2_dist_all <= radius) + 1e-10) flag = 1 if (l1_density > 0.21 or l2_density > 0.21) and radius > 12 and points_num > 60 else 0 type = 'Maniflod' if flag == 0 else 'HyperSphere' print(f'Distribution: {flag} / {type}.') output_path = './results/judge/fft/' + str(flag) + '/' + name + '.png' img_up = np.hstack([img_, np.repeat(magnitude, 3).reshape((height, width, 3))]) os.makedirs(os.path.dirname(output_path), exist_ok=True) cv2.imwrite(output_path, img_up) return flagdef create_storage_folder(
main_folder_path, project_folder, group_folder, run_name, mode=”iterate”
):
os.makedirs(main_folder_path, exist_ok=True)
save_path = main_folder_path
return save_pathdef set_torch_device(gpu_ids):
“””Returns correct torch.device.Args: gpu_ids: [list] list of gpu ids. If empty, cpu is used. """ if torch.backends.mps.is_available(): return torch.device("mps") elif len(gpu_ids): return torch.device("cuda:{}".format(gpu_ids[0])) return torch.device("cpu")def fix_seeds(seed, with_torch=True, with_cuda=True):
“””Fixed available seeds for reproducibility.Args: seed: [int] Seed value. with_torch: Flag. If true, torch-related seeds are fixed. with_cuda: Flag. If true, torch+cuda-related seeds are fixed """ random.seed(seed) np.random.seed(seed) if with_torch: torch.manual_seed(seed) if with_cuda and torch.cuda.is_available(): torch.cuda.manual_seed(seed) torch.cuda.manual_seed_all(seed) torch.backends.cudnn.deterministic = Truedef compute_and_store_final_results(
results_path,
results,
column_names,
row_names=None,
):
“””Store computed results as CSV file.Args: results_path: [str] Where to store result csv. results: [List[List]] List of lists containing results per dataset, with results[i][0] == 'dataset_name' and results[i][1:6] =[instance_auroc, full_pixelwisew_auroc, full_pro,
anomaly-only_pw_auroc, anomaly-only_pro]“”” if row_names is not None: assert len(row_names) == len(results), “#Rownames != #Result-rows.” mean_metrics = {} for i, result_key in enumerate(column_names): mean_metrics[result_key] = np.mean([x[i] for x in results]) savename = os.path.join(results_path, “results.csv”) with open(savename, “w”) as csv_file: csv_writer = csv.writer(csv_file, delimiter=”,”) header = column_names if row_names is not None: header = [“Row Names”] + header csv_writer.writerow(header) for i, result_list in enumerate(results): csv_row = result_list if row_names is not None: csv_row = [row_names[i]] + result_list csv_writer.writerow(csv_row) mean_scores = list(mean_metrics.values()) if row_names is not None: mean_scores = [“Mean”] + mean_scores csv_writer.writerow(mean_scores) mean_metrics = {“mean_{0}”.format(key): item for key, item in mean_metrics.items()} return mean_metrics
def del_remake_dir(path, del_flag=True):
if os.path.exists(path):
if del_flag:
shutil.rmtree(path, ignore_errors=True)
os.makedirs(path, exist_ok=True)
else:
os.makedirs(path, exist_ok=True)def torch_format_2_numpy_img(img):
if img.shape[0] == 3:
img = img.transpose([1, 2, 0])
img = img * np.array(IMAGENET_STD) + np.array(IMAGENET_MEAN)
img = img[:, :, [2, 1, 0]]
img = (img * 255).astype(‘uint8’)
else:
img = img.transpose([1, 2, 0])
img = np.repeat(img, 3, axis=-1)
img = (img * 255).astype(‘uint8’)
return img
shell içerisindeki run-mvtech.sh içeriğim ise aşağıdaki gibi modifiye edildi:
datapath=/Users/veyselakbas/Desktop/glass_project/glass/datasets/
augpath=/Users/veyselakbas/Desktop/glass_project/dtd/images
classes=(‘carpet’ ‘grid’ ‘leather’ ’tile’ ‘wood’ ‘bottle’ ‘cable’ ‘capsule’ ‘hazelnut’ ‘metal_nut’ ‘pill’ ‘screw’ ‘toothbrush’ ‘transistor’ ‘zipper’)
flags=($(for class in “${classes[@]}”; do echo ‘-d ‘”${class}”; done))cd /Users/veyselakbas/Desktop/glass_project/glass
python main.py \
–gpu 0 \
–seed 0 \
–test ckpt \
net \
-b wideresnet50 \
-le layer2 \
-le layer3 \
–pretrain_embed_dimension 1536 \
–target_embed_dimension 1536 \
–patchsize 3 \
–meta_epochs 640 \
–eval_epochs 1 \
–dsc_layers 2 \
–dsc_hidden 1024 \
–pre_proj 1 \
–mining 1 \
–noise 0.015 \
–radius 0.75 \
–p 0.5 \
–step 20 \
–limit 392 \
dataset \
–distribution 0 \
–mean 0.5 \
–std 0.1 \
–fg 1 \
–rand_aug 1 \
–batch_size 8 \
–resize 288 \
–imagesize 288 “${flags[@]}” mvtec $datapath $augpath
3. MPS ile GLASS Modelini Çalıştırma
GLASS modelini çalıştırmadan önce MPS’in uygun şekilde yapılandırıldığından emin olun. Ardından, eğitim ve test süreçlerini başlatabilirsiniz:
Eksik modülleri yükleyelim
pip install imgaug
ardından
pip install openpyxl
Maksimum açık dosya sayısı problemi yaşamamak için aşağıdaki adımları uygulayın
Mac İçin Özel Ayarlar:
MacOS’ta açık dosya sınırını artırmak için sistem ayarlarını düzenlemeniz gerekebilir:
- 1. /etc/sysctl.conf dosyasını düzenleyin (yoksa oluşturun):
sudo nano /etc/sysctl.conf
- 2. Aşağıdaki satırları ekleyin:
kern.maxfiles=11165536
kern.maxfilesperproc=11165536
- 3. Değişiklikleri uygulayın:
sudo sysctl -w kern.maxfiles=11165536
sudo sysctl -w kern.maxfilesperproc=11165536
- 4. /etc/launchd.conf dosyasını düzenleyin (yoksa oluşturun):
sudo nano /etc/launchd.conf
- 5. Aşağıdaki satırı ekleyin:
limit maxfiles 11165536 11165536
- 6. Sistemi yeniden başlatın.
glass içerisindeki shell içerisindeki run-mvtec.sh dosyasına önce gerekli izinleri verelim shell içerisine girelim.
cd shell
Çalıştırma iznini verelim
chmod +x run-mvtec.sh
Çalıştıralım
./run-mvtec.sh
7. Potansiyel Sorunlar ve Çözümleri
• MPS Backend’in Desteklemediği Operasyonlar: PyTorch’un MPS backend’i bazı özel operasyonları desteklemeyebilir. Bu durumda hata alırsanız, ilgili kısımları cpu üzerinde çalıştırabilirsiniz:
outputs = some_function(inputs.to(“cpu”))
• Performans Testi: MacOS’un “Activity Monitor” uygulamasında GPU sekmesini kontrol ederek GPU kullanımını doğrulayabilirsiniz.
Bu yönergeleri takip ederek GLASS projesini M1 Max GPU’nuzda çalıştırabilirsiniz. Sorularınız veya ek ihtiyaçlarınız olursa çekinmeden sorabilirsiniz!
